Introduction I began my career in electrical engineering in 1969 when I graduated from NYU’s school of engineering and science. At that time, the best that the school could afford were a collection of vacuum tube laboratory systems, DC motors and a single large IBM mainframe computer. Students could practice programming through the creation of punch card decks with one programming instruction per card and therefore, a program was typically represented by a stack of cards often several feet high. It’s fun to recall that special features were added so that if you accidentally dropped your program on the floor, and shuffled the cards, you could reorder the deck though the use of a sorting machine that was the size of several washing machines.
Clearly, technology has come a long way. In one of his recent books called “The Singularity” Ray Kurzweil, the popular futurist and entrepreneur speculates about the ever increasing speed with which technology seems to move. His point is that technology feeds upon itself, the latest computer is used to design the next computer, and so on, making the rate of change exponential as opposed to a linear progression of advancement. We are taught how the progressive doubling of some quantity can quickly build into an enormous number through the simple example of attempting to repeatedly fold a piece of paper, doubling its thickness with each fold. After a mere 17 folds (if one could do it) the paper would be taller than the average house, and after 50 folds would reach to the sun!
For the duration of my career, I have been involved in the digital imaging industry and the effect of this exponential growth can be illustrated over the last forty years by observing how this one field of technology has evolved.
In the early 1970s, digital imaging computers were referred to as signal processors or digital signal processors, and eventually shortened to the DSP. The only applications that one could find back then were related to the military because of the prohibitive cost of these computers. One of the first projects that I worked on was an “over the horizon” radar system that would watch for enemy intercontinental ballistic missiles (ICBMs). The radar data would be processed by a room full, over 300 cubic feet, of computer hardware that would create images that could be viewed and evaluated by an operator. This multi-computer system would accomplish something in the order of around 6 million arithmetic multiplies per second.
In 1965, Gordon E. Moore, cofounder of Intel, predicted quite accurately that digital electronic circuits would double their density approximately every two years. So it was fairly soon into the 1970s that these same mathematical algorithms used for radar image processing were finding their way into the first commercial market application, medical imaging, specifically in the form of CAT scanners. The specific mathematical equations that were being used to process radar images were similar to those needed to reconstruct x-ray images as done within a CAT scanner but the price of this equipment and its size were dropping rapidly as predicted by Moore. Early CAT scanners could be purchased for 1 to 2 million dollars.
In the 1980s it became affordable to bring the same mathematical processes to yet a new market, graphic arts imaging, in the form of computers that would help an artist or designer lay out a magazine or newspaper page on a computer screen in place of the manual techniques which would require a razor blade and film. These systems had again dropped in price again, breaking into a new application.
It is interesting to note that the basic science of image processing was not really changing that much. For example, Johannes Kepler, a German astronomer and mathematician had defined the equations that would be used in a CAT scanner back in the early 1600s. What was changing was that with each progressive cycle of redesign of the basic hardware, its speed, size and cost were all rapidly improving and frequently surging through market opportunities that became practical when cost performance objectives could be achieved.
From graphic arts imaging, basic digital imaging techniques were next found in consumer imaging, most famously in the form of digital cameras that today can be found in virtually every cell phone of which there are billions already in use.
From a room full of computer equipment costing millions of dollars, today the digital camera portion of a cell phone can cost in the range of tens of dollars, and yet exceed the performance of their early forbearers by a factor of 30 times in speed. A common cell phone might perform mathematical multiplies at a rate of 1 GHz or 1 billion per second. This is an improvement of cost and performance of roughly 1000 times overall in 40 years.
From a generational perspective, we might consider the DSP as having gone through four generations from military, to medical, to graphic arts, and finally consumer applications. And now we stand that the edge of the fifth generation which I call surrogacy.
Surrogacy
The concept of the imaging surrogacy market is that cameras in combination with small fast computers are capable of watching and making autonomous decisions based upon what is being seen. Those decisions can be communicated to other systems, or can, for example, in a simple case, activate a machine. As in prior markets, the application of imaging to the surrogacy market is not a question of “if it will occur” but rather, more simply, “when it will occur,” and current technological advancements have brought us to that moment now.
It was estimated by the automotive industry that over the next 10 years, one would find tens if not hundreds of digital cameras in every automobile. But most would not be used in the way that one might expect. It’s true that you would find low cost digital cameras used in place or as an adjunct to mirrors to assist the driver in seeing otherwise hidden vantage points. Back-up cameras are common in many vehicles. But this is a provincial use of digital imaging and the real volume of applications are represented by imaging surrogacy systems.
It has been common to think of low cost imaging as simply a way to bring images to a human operator at a remote location. For example, a security camera in a store delivers its images to a console in the security office where a guard can scan and watch for thieves.
However, the concept of surrogacy implies that the camera along with its DSP not only watches and potentially records what is seen, but more importantly it can make fundamental decisions, autonomously, without the need for human intervention.
So, for example, the pressure switch that might be located in the passenger seat that would disable a passenger airbag from being deployed when a child is sitting there would be replaced by an inexpensive digital camera and DSP that would “see” the passenger, measure his/her size and position and make an airbag deployment decision based upon the processed result.
In automotive applications, it is anticipated that cameras will be used to observe and make decisions on: • Operator fatigue by watching the head and eyes of the operator, • Lane drifting by watching the dashed or solid lines, • Distance to an obstruction by watching (often in stereo) objects in front and behind the car • Dimming the headlamps by watching for an approaching vehicle • Deploying the airbag (as described) • Obtaining the speed limit by reading signs • Observing danger by reading signs • Steering by observing the road • Measuring the speed by observing the road • Watching for rain and dirt
As well, insurance companies are deploying automotive cameras that record the last few minutes or seconds before an accident which can be used to assign blame or improve designs.
Today Volvo is testing a car which contains cameras that automatically avoid collisions and accidents by seeing and processing the objects and more importantly pedestrians in the path of the car.
As a completely different example that is already in common use, fully autonomous systems can already be found in Las Vegas gambling casinos watching for the faces of known card counters (or cheats from the perspective of the casinos) so that they can be escorted from the property.
As the cost of these systems continues to drop, it is only more likely that they permeate our environment as sensors that are used to make basic decisions like: should the lights and heat (or A/C) be on based upon whether there is any person in the room, or should the brakes be applied based upon whether there is an obstruction in the path of a car.
One can also envision imaging systems that are hybrid in nature and include human operators for whom the responsibility of decisions is aided by the processor through either the reduction of information to only relevant images, or through the enhancement or highlighting of suspected regions of interest. A common example today is the highlighting of mammography images to aid a radiologist in finding micro-calcifications, an early sign of cancer. Other examples include: • Security cameras that detect motion automatically and pass video clips of interest to an operator. • Security cameras that can discriminate between a dog that has been intentionally left on the property from an intruder who should not be there.
One must consider the application of surrogacy cameras from the perspective of: “if I could put a tireless human at this position and ask them to watch for a simple (or in some cases, not so simple) event at virtually no cost and with no environmental or size limitations, would I choose to do so?” Such cameras can be mounted on the wings of airplanes looking for “out of the ordinary” changes, or on cruise ships looking for an intoxicated passenger who accidentally falls overboard. These cameras have the advantages of never becoming tired, or bored. They can see in a variety of spectra, for example IR or UV, and can watch and observe endlessly.
Soon, cameras along with their autonomous computers will be less than a few dollars each which will cause an explosion of these types of applications.
Finding Surrogacy Solutions
Finding surrogacy applications begins with forgetting commonly held beliefs about the use of cameras in industrial or consumer environments.
The key is to identify events, which if observed, have a direct cost or profit impact on one’s business. Often, a safety or theft issue can drive the need for continuous observation, processing and a resulting decision that in some cases, calls for the attention of a human and in other cases makes a decision relating to the operation of the system.
Keep in mind that these cameras can be tiny, especially if used in higher quantities or for applications where high value is obtained. In these cases, one can integrate the camera and processor into a single tiny package, often no larger than a pencil eraser head. Systems that are this small are often easier to hide or protect environmentally. Today, there are complete wireless cameras that can be swallowed in order to inspect the patient’s GI tract.
Camera systems can be completely encased in plastic thereby eliminating many environmental considerations. Camera systems can provide mono or stereo vision and, like humans, can therefore make accurate distance assessments based upon a stereo pair. Camera systems can observe changes over time periods that are too slow or fast for human observation.
For me, it’s fun to ponder some future applications. Here are some of my favorites. All of these are possible within the context of currently available hardware and software. • A lapel camera and earphone that reminds me of the name of an approaching customer at a trade show. Imagine this used in retail environments to remind the sales agent of the identity and history of a shopper that enters the store. • A door camera that unlocks my door when I or my family approach. • A light switch and thermostat camera that saves money when no one is in the room. • Cameras used in retail environments that watch for theft by observing that objects are moved past the register without properly being scanned. Or taken by a customer without being paid for. • Cameras on complicated machines like airplanes and boats that call attention to malfunctions so as to create a faster appreciation for the situational emergency. • Cameras at train stations that stop trains if tracks are obstructed by a fallen passenger. • Cameras that watch for fires or leaks or floods. • Cameras that protect hands and fingers from machinery. • Cameras that scan skin for cancerous lesions.
It’s endless.