Plant-e: heats, shoots and leaves — electricity from living plants

— the story —

Plants could soon provide our electricity. In a small way they already are doing that in research labs and greenhouses at project Plant-e — a university and commercially sponsored research group at Wageningen University in the Netherlands.

The Plant Microbial Fuel Cell from Plant-e can generate electricity from the natural interaction between plant roots and soil bacteria. It works by taking advantage of the up to 70 percent of organic material produced by a plant’s photo-synthesis process that cannot be used by the plant — and is excreted through the roots.

As natural occurring bacteria around the roots break down this organic residue, electrons are released as a waste product. By placing an electrode close to the bacteria to absorb these electrons, the research team — led by Marjolein Helder PhD — is able to generate electricity.

Helder said: “Solar panels are making more energy per square meter — but we expect to reduce the costs of our system technology in the future. And our system can be used for a variety of applications.”

Plant Microbial Fuel Cells can be used on many scales. An experimental 15 square meter model can produce enough energy to power a computer notebook. Plant-e is working on a system for large scale electricity production in existing green areas like wetlands and rice paddy fields.

Helder said: “Our technology is making electricity — but also could be used as roof insulation or as a water collector. On a bigger scale it’s possible to produce rice and electricity at the same time, and in that way combine food and energy production.”

A first prototype of a green electricity roof has been installed on one building at Wageningen University and researchers are keeping a close eye on what is growing there. The first field pilots will be started in 2014. The technology was patented in 2007.

After 5 years of lab research: Plant-e is now taking the first steps toward commercializing the technology. In the future, bio-electricity from plants could produce as much as 3.2 watts per square meter of plant growth.

w. descriptions from: EuroNews

Plant-e | main
Plant-e | brochure
Plant-e | YouTube channel

video | electricity from plants

— watch • videos from Plant-e —

Plant-e | video: animation
Plant-e | video: the power of plants
Plant-e | video: the next step in development

on the web | essentials

Wageningen Univ. | main
Wageningen Univ. | research institutes: plant research
Wageningen Univ. | research institutes: centre for development innovation
Wageningen Univ. | story: Dutch Innovation Award for Plant-e

subscribe: for our weekly digest

Dear readers,

Sign up for our free newsletter, published twice weekly. We assure our readers that your information is safely managed according to the new GDPR regulations.

We will soon be upgrading the account subscription section sign-up and account management tool, and adding new features to the newsletter. This will simplify your ability to change your subscribed e-mail and other account settings.

  • If you’ve been subscribed to our weekly newsletter, that will now arrive twice weekly.
  • If you’re subscribed to the daily newsletter, that will also arrive twice weekly.

Our summer season publishing schedule is Tuesday + Friday evenings.

We’ll be updating the subscription account section to merge both daily and weekly subscriptions into this new, simpler format and upgrading with some new tools — to improve your reader experience.

For our readers who’ve asked about a mobile version of this website, we’re still researching that and we do understand that the lower menu is not tablet device or smart phone compatible. As we continue to upgrade over the summer, we’ll be looking into smaller format reading for our subscribers hoping to read on iPad, mobile phones or other tablets.

Thank you for reading and enjoy!

managing editor

Nanomaterials that mimic nerve impulses (spikes) discovered

Nanomaterials that mimic nerve impulses (credit: Osaka University)

A combination of nanomaterials that can mimic nerve impulses (“spikes”) in the brain have been discovered by researchers at Kyushu Institute of Technology and Osaka University in Japan.

Current “neuromorphic” (brain-like) chips (such as IBM’s neurosynaptic TrueNorth) and circuits (such as those based on the NVIDIA GPGPU, or general purpose graphical processing unit) are devices based on complex circuits that emulate only one part of the brain’s mechanisms: the learning ability of synapses (which connect neurons together).

(Left) Schematic of the SWNT/POM complex network, showing single-wall nanotubes and polyoxometalate (POM) molecules, with gold contacts. (Right) Conductive atomic force microscope image of a molecular neuromorphic network device. (Inset) Molecular structure of polyoxometalate (POM) molecules. (credit: Hirofumi Tanaka et al./Nature Communications)

The researchers have now developed a way to simulate a large-scale spiking neural network. They created a complex SWNT/POM molecular neuromorphic device consisting of a dense and complex network of spiking molecules. The new nanomaterial comprises polyoxometalate (POM) molecules that are absorbed by single-wall carbon nanotubes (SWNTs).

Unlike ordinary organic molecules, POM consists of metal atoms and oxygen atoms that form a three-dimensional framework that can store charges in a single molecule. The new nanomaterial emits spikes and can transmit them via synapses to and from other neurons.

The researchers also demonstrated that this molecular model could be used as a component of reservoir computing devices, which are anticipated as next-generation neural network devices.

Ref: Nature Communications (open access). Source: Osaka University

How to predict the side effects of millions of drug combinations

An example graph of polypharmacy side effects derived from genomic and patient population data, protein–protein interactions, drug–protein targets, and drug–drug interactions encoded by 964 different polypharmacy side effects. The graph representation is used to develop Decagon. (credit: Marinka Zitnik et al./Bioinformatics)

Millions of people take up to five or more medications a day, but doctors have no idea what side effects might arise from adding another drug.*

Now, Stanford University computer scientists have developed a deep-learning system (a kind of AI modeled after the brain) called Decagon** that could help doctors make better decisions about which drugs to prescribe. It could also help researchers find better combinations of drugs to treat complex diseases.

The problem is that with so many drugs currently on the U.S. pharmaceutical market, “it’s practically impossible to test a new drug in combination with all other drugs, because just for one drug, that would be five thousand new experiments,” said Marinka Zitnik, a postdoctoral fellow in computer science and lead author of a paper presented July 10 at the 2018 meeting of the International Society for Computational Biology.

With some new drug combinations (“polypharmacy”), she said, “truly we don’t know what will happen.”

How proteins interact and how different drugs affect these proteins

So Zitnik and associates created a network describing how the more than 19,000 proteins in our bodies interact with each other and how different drugs affect these proteins. Using more than 4 million known associations between drugs and side effects, the team then designed a method to identify patterns in how side effects arise, based on how drugs target different proteins, and also to infer patterns about drug-interaction side effects.***

Based on that method, the system could predict the consequences of taking two drugs together.

To evaluate the The research was supported by the National Science Foundation, the National Institutes of Health, the Defense Advanced Research Projects Agency, the Stanford Data Science Initiative, and the Chan Zuckerberg Biohub. system, the group looked to see if its predictions came true. In many cases, they did. For example, there was no indication in the original data that the combination of atorvastatin (marketed under the trade name Lipitor among others), a cholesterol drug, and amlopidine (Norvasc), a blood-pressure medication, could lead to muscle inflammation. Yet Decagon predicted that it would, and it was right.

In the future, the team members hope to extend their results to include more multiple drug interactions. They also hope to create a more user-friendly tool to give doctors guidance on whether it’s a good idea to prescribe a particular drug to a particular patient, and to help researchers developing drug regimens for complex diseases, with fewer side effects.

Ref.: Bioinformatics (open access). Source: Stanford University.

* More than 23 percent of Americans took three or more prescription drugs in the past 30 days, according to a 2017 CDC estimate. Furthermore, 39 percent over age 65 take five or more, a number that’s increased three-fold in the last several decades. There are about 1,000 known side effects and 5,000 drugs on the market, making for nearly 125 billion possible side effects between all possible pairs of drugs. Most of these have never been prescribed together, let alone systematically studied, according to the Stanford researchers.

** In geometry, a decagon is a ten-sided polygon.

*** The research was supported by the National Science Foundation, the National Institutes of Health, the Defense Advanced Research Projects Agency, the Stanford Data Science Initiative, and the Chan Zuckerberg Biohub.


MIT’s Cheetah 3 blind robot can climb a staircase littered with debris, leap, and gallop across rough terrain

MIT’s Cheetah 3 robot — an upgrade to the Cheetah 2, can now leap and gallop across rough terrain, climb a staircase littered with debris, and quickly recover its balance when suddenly yanked or shoved — all while essentially blind.

The 90-pound robot is intentionally designed to do all this without relying on cameras or any external environmental sensors. The idea is to allow it to “feel” its way through its surroundings via “blind locomotion,” (like making your way across a pitch-black room), eliminating visual distractions, which would slow the robot down.

“Vision can be noisy, slightly inaccurate, and sometimes not available, and if you rely too much on vision, your robot has to be very accurate in position and eventually will be slow, said the robot’s designer, Sangbae Kim, associate professor of mechanical engineering at MIT. “So we want the robot to rely more on tactile information. That way, it can handle unexpected obstacles while moving fast.”

Faster, more nimble, more cat-like

Warning: Cheetah 3 can jump on your desk (credit: MIT)

Cheetah3 has an expanded range of motion compared to its predecessor Cheetah 2, which allows the robot to stretch backwards and forwards, and twist from side to side, much like a cat limbering up to pounce. Cheetah 3 can blindly make its way up staircases and through unstructured terrain, and can quickly recover its balance in the face of unexpected forces, thanks to two new algorithms developed by Kim’s team: a contact detection algorithm, and a model-predictive control algorithm.

The contact detection algorithm helps the robot determine the best time for a given leg to switch from swinging in the air to stepping on the ground. For example, if the robot steps on a light twig versus a hard, heavy rock, how it reacts — and whether it continues to carry through with a step, or pulls back and swings its leg instead — can make or break its balance.

The researchers tested the algorithm in experiments with the Cheetah 3 trotting on a laboratory treadmill and climbing on a staircase. Both surfaces were littered with random objects such as wooden blocks and rolls of tape.

The robot’s blind locomotion was also partly due to the model-predictive control algorithm, which predicts how much force a given leg should apply once it has committed to a step. The model-predictive control algorithm calculates the multiplicative positions of the robot’s body and legs a half-second into the future, if a certain force is applied by any given leg as it makes contact with the ground.

Cameras to be activated later

The team had already added cameras to the robot to give it visual feedback of its surroundings. This will help in mapping the general environment, and will give the robot a visual heads-up on larger obstacles such as doors and walls. But for now, the team is working to further improve the robot’s blind locomotion

“We want a very good controller without vision first,” Kim says. “And when we do add vision, even if it might give you the wrong information, the leg should be able to handle [obstacles]. Because what if it steps on something that a camera can’t see? What will it do? That’s where blind locomotion can help. We don’t want to trust our vision too much.”

Within the next few years, Kim envisions the robot carrying out tasks that would otherwise be too dangerous or inaccessible for humans to take on.

This research was supported, in part, by Naver, Toyota Research Institute, Foxconn, and Air Force Office of Scientific Research.

Source: MIT.

Discovering new drugs and materials by ‘touching’ molecules in virtual reality

To figure out how to block a bacteria’s attempt to create multi-resistance to antibiotics, a researcher grabs a simulated ligand (binding molecule) — a type of penicillin called benzylpenicillin (red) — and interactively guides that molecule to dock within a larger enzyme molecule (blue-orange) called β-lactamase, which is produced by bacteria in an attempt to disable penicillin (making a patient resistant to a class of antibiotics called β-lactam). (credit: University of Bristol)

University of Bristol researchers have designed and tested a new virtual reality (VR) cloud-based system intended to allow researchers to reach out and “touch” molecules as they move — folding them, knotting them, plucking them, and changing their shape to test how the molecules interact. Using an HTC Vive virtual-reality device, it could lead to creating new drugs and materials and improving the teaching of chemistry.

More broadly, the goal is to accelerate progress in nanoscale molecular engineering areas that include conformational mapping, drug development, synthetic biology, and catalyst design.

Real-time collaboration via the cloud

Two users passing a fullerene (C60) molecule back and forth in real time over a cloud-based network. The researchers are each wearing a VR head-mounted display (HMD) and holding two small wireless controllers that function as atomic “tweezers” to manipulate the real-time molecular dynamic of the C60 molecule. Each user’s position is determined using a real-time optical tracking system composed of synchronized infrared light sources, running locally on a GPU-accelerated computer. (credit: University of Bristol)

The multi-user system, developed by developed by a team led by University of Bristol chemists and computer scientists, uses an “interactive molecular dynamics virtual reality” (iMD VR) app that allows users to visualize and sample (with atomic-level precision) the structures and dynamics of complex molecular structures “on the fly” and to interact with other users in the same virtual environment.

Because each VR client has access to global position data of all other users, any user can see through his/her headset a co-located visual representation of all other users at the same time. So far, the system has uniquely allowed for simultaneously co-locating six users in the same room within the same simulation.

Testing on challenging molecular tasks

The team designed a series of molecular tasks for testing, using traditional mouse, keyboard, and touchscreens compared to virtual reality. The tasks included threading a small molecule through a nanotube, changing the screw-sense of a small organic helix, and tying a small string-like protein into a simple knot, and a variety of dynamic molecular problems, such as binding drugs to its target, protein folding, and chemical reactions. The researchers found that for complex 3D tasks, VR offers a significant advantage over current methods. For example, participants were ten times more likely to succeed in difficult tasks such as molecular knot tying.

Anyone can try out the tasks described in the open-access paper by downloading the software and launching their own cloud-hosted session.

David Glowacki | This video, made by University of Bristol PhD student Helen M. Deeks, shows the actions she took using a wireless set of “atomic tweezers” (using the HTC Vive) to interactively dock a single benzylpenicillin drug molecule into the active site of the β-lactamase enzyme. 

David Glowacki | The video shows the cloud-mounted virtual reality framework, with several different views overlaid to give a sense of how the interaction works. The video outlines the four different parts of the user studies: (1) manipulation of buckminsterfullerene, enabling users to familarize themselves with the interactive controls; (2) threading a methane molecule through a nanotube; (3) changing the screw-sense of a helicene molecule; and (4) tying a trefoil knot in 17-Alanine.

Ref: Science Advances (open-access). Source: University of Bristol.

There’s no known upper limit to human longevity, study suggests

Chiyo Miyako of Japan is the world’s oldest verified living person at 117 years, as of June 29, 2018, according to the Gerontology Research Group. She credits eating eel, drinking red wine, and never smoking for her longevity, and enjoys calligraphy. (credit: Medical Review Co., Ltd.)

Human death risk increases exponentially from 65 up to about age 80. At that point, the range of risks starts to increase. But by age 105, the death risk actually levels off — suggesting there’s no known upper limit for human lifespan.*

That’s the conclusion of a controversial study by an international team of scientists, published Thursday, June 28 in the journal Science.

“The increasing number of exceptionally long-lived people and the fact that their mortality beyond 105 is seen to be declining across cohorts — lowering the mortality plateau or postponing the age when it appears — strongly suggest that longevity is continuing to increase over time and that a limit, if any, has not been reached,” the researchers wrote.

High-quality data

Logarithmic plot of the exponential risk of death (“hazard”) from ages 65 to 115 (on a logarithmic plot, exponentials are shown as a diagonal straight line). For ages up to 105, the data is from the Human Mortality Database (HMD). Note that starting at age 80, the range of risks of death (blue bars) starts to increase (people live to different ages; some live longer) — it’s no longer a fixed probability, as in the traditional “Gompertz” model (black line). However by age 105, based on data from the new Italian ISTAT model, the risk of death actually hits a plateau (stops increasing, as shown in dashed black line with orange background) and the odds of someone dying from one birthday to the next are roughly 50:50.** (credit: E. Barbi et al., Science)

The new study was based on “high-quality data from Italians aged 105 and older, collected by the Italian National Institute of Statistics (ISTAT).” That data provided “accuracy and precision that were not possible before,” the researchers say.

Previous data for supercentenarians (age 110 or older) in the Max Planck Institute for Demographic Research International Database on Longevity (IDL) were “problematic in terms of age reporting [due to] sparse data pooled from 11 countries,” according to the authors of the Science paper.

Instead, ISTAT “collected and validated the individual survival trajectory” of all inhabitants of Italy aged 105 and older in the period from 1 January 2009 to 31 December 2015, including birth certificates, suggesting that “misreporting is believed to be minimal in these data.”

Ref.: Science.

* The current record for the longest human life span was set in 1997 when French woman Jeanne Calment died at the age of 122.

** This chart shows yearly hazards (probability of death) on a logarithmic scale for the cohort (group of subjects with a matching characteristic) of Italian women born in 1904. The straight-line prediction (black) is based on fitting a Gompertz model to ages 65 to 80. Confidence intervals (blue) — the range of death probabilities — were derived from Human Mortality Database (HMD) data for ages up to 105, and from ISTAT data beyond age 105. Note the longer intervals and increased diversion from straight-line prediction (black) after age 80; and the estimated plateau in probability of death values beyond age 105 (black dashed line with orange background), based on the model parameters, which were in turn based on the full ISTAT database.

New material eliminates need for motors or actuators in future robots, other devices

A “mini arm” made up of two hinges of actuating nickel hydroxide-oxyhydroxide material (left) can lift an object 50 times its own weight when triggered (right) by light or electricity. (credit: University of Hong Kong)

University of Hong Kong researchers have invented a radical new lightweight material that could replace traditional bulky, heavy motors or actuators in robots, medical devices, prosthetic muscles, exoskeletons, microrobots, and other types of devices.

The new actuating material — nickel hydroxide-oxyhydroxide — can be instantly triggered and wirelessly powered by low-intensity visible light or electricity at relatively low intensity. It can exert a force of up to 3000 times its own weight — producing stress and speed comparable to mammalian skeletal muscles, according to the researchers.

The material is also responsive to heat and humidity changes, which could allow autonomous machines to harness tiny energy changes in the environment.

The major component is nickel, so the material cost is low, and fabrication uses a simple electrodeposition process, allowing for scaling up and manufacture in industry.

Developing actuating materials was identified as the leading grand challenge in “The grand challenges of Science Robotics” to “deeply root robotics research in science while developing novel robotic platforms that will enable new scientific discoveries.”

Using a light blocker (top) a mini walking bot (bottom) with the “front leg” bent and straightened alternatively can walk towards a light source. (credit: University of Hong Kong)

University of Hong Kong | Future Robots need No Motors

Ref.: Science Robotics. Source: University of Hong Kong.

How robots aided by deep learning could help autism therapists

MIT Media Lab  (no sound) | Intro: Personalized Machine Learning for Robot Perception of Affect and Engagement in Autism Therapy. This is an example of a therapy session augmented with SoftBank Robotics’ humanoid robot NAO and deep-learning software. The 35 children with autism who participated in this study ranged in age from 3 to 13. They reacted in various ways to the robots during their 35-minute sessions — from looking bored and sleepy in some cases to jumping around the room with excitement, clapping their hands, and laughing or touching the robot.

Robots armed with personalized “deep learning” software could help therapists interpret behavior and personalize therapy of autistic children, while making the therapy more engaging and natural. That’s the conclusion of a study by an international team of researchers at MIT Media Lab, Chubu University, Imperial College London, and University of Augsburg.*

Children with autism-spectrum conditions often have trouble recognizing the emotional states of people around them — distinguishing a happy face from a fearful face, for instance. So some therapists use a kid-friendly robot to demonstrate those emotions and to engage the children in imitating the emotions and responding to them in appropriate ways.

Personalized autism therapy

But the MIT research team realized that deep learning would help the therapy robots perceive the children’s behavior more naturally, they report in a Science Robotics paper.

Personalization is especially important in autism therapy, according to the paper’s senior author, Rosalind Picard, PhD, a professor at MIT who leads research in affective computing: “If you have met one person, with autism, you have met one person with autism,” she said, citing a famous adage.

“Computers will have emotional intelligence by 2029”… by which time, machines will “be funny, get the joke, and understand human emotion.” — Ray Kurzweil

“The challenge of using AI [artificial intelligence] that works in autism is particularly vexing, because the usual AI methods require a lot of data that are similar for each category that is learned,” says Picard, in explaining the need for deep learning. “In autism, where heterogeneity reigns, the normal AI approaches fail.”

How personalized robot-assisted therapy for autism would work

Robot-assisted therapy** for autism often works something like this: A human therapist shows a child photos or flash cards of different faces meant to represent different emotions, to teach them how to recognize expressions of fear, sadness, or joy. The therapist then programs the robot to show these same emotions to the child, and observes the child as she or he engages with the robot. The child’s behavior provides valuable feedback that the robot and therapist need to go forward with the lesson.

“Therapists say that engaging the child for even a few seconds can be a big challenge for them. [But] robots attract the attention of the child,” says lead author Ognjen Rudovic, PhD, a postdoctorate fellow at the MIT Media Lab. “Also, humans change their expressions in many different ways, but the robots always do it in the same way, and this is less frustrating for the child because the child learns in a very structured way how the expressions will be shown.”

SoftBank Robotics | The researchers used NAO humanoid robots in this study. Almost two feet tall and resembling an armored superhero or a droid, NAO conveys different emotions by changing the color of its eyes, the motion of its limbs, and the tone of its voice.

However, this type of therapy would work best if the robot could also smoothly interpret the child’s own behavior — such as excited or paying attention — during the therapy, according to the researchers. To test this assertion, researchers at the MIT Media Lab and Chubu University developed a personalized deep learning network that helps robots estimate the engagement and interest of each child during these interactions, they report.**

The researchers built a personalized framework that could learn from data collected on each individual child. They captured video of each child’s facial expressions, head and body movements, poses and gestures, audio recordings and data on heart rate, body temperature, and skin sweat response from a monitor on the child’s wrist.

Most of the children in the study reacted to the robot “not just as a toy but related to NAO respectfully, as it if was a real person,” said Rudovic, especially during storytelling, where the therapists asked how NAO would feel if the children took the robot for an ice cream treat.

In the study, the researchers found that the robots’ perception of the children’s responses agreed with assessments by human experts with a high correlation score of 60 percent, the scientists report.*** (It can be challenging for human observers to reach high levels of agreement about a child’s engagement and behavior. Their correlation scores are usually between 50 and 55 percent, according to the researchers.)

Ref.: Science Robotics (open-access). Source: MIT

* The study was funded by grants from the Japanese Ministry of Education, Culture, Sports, Science and Technology; Chubu University; and the European Union’s HORIZON 2020 grant (EngageME).

** A deep-learning system uses hierarchical, multiple layers of data processing to improve its tasks, with each successive layer amounting to a slightly more abstract representation of the original raw data. Deep learning has been used in automatic speech and object-recognition programs, making it well-suited for a problem such as making sense of the multiple features of the face, body, and voice that go into understanding a more abstract concept such as a child’s engagement.

Overview of the key stages (sensing, perception, and interaction) during robot-assisted autism therapy.
Data from three modalities (audio, visual, and autonomic physiology) were recorded using unobtrusive audiovisual sensors and sensors worn on the child’s wrist, providing the child’s heart-rate, skin-conductance (EDA), body temperature, and accelerometer data. The focus of this work is the robot perception, for which we designed the personalized deep learning framework that can automatically estimate levels of the child’s affective states and engagement. These can then be used to optimize the child-robot interaction and monitor the therapy progress (see Interpretability and utility). The images were obtained by using Softbank Robotics software for the NAO robot. (credit: Ognjen Rudovic et al./Science Robotics)

“In the case of facial expressions, for instance, what parts of the face are the most important for estimation of engagement?” Rudovic says. “Deep learning allows the robot to directly extract the most important information from that data without the need for humans to manually craft those features.”

The robots’ personalized deep learning networks were built from layers of these video, audio, and physiological data, information about the child’s autism diagnosis and abilities, their culture and their gender. The researchers then compared their estimates of the children’s behavior with estimates from five human experts, who coded the children’s video and audio recordings on a continuous scale to determine how pleased or upset, how interested, and how engaged the child seemed during the session.

*** Trained on these personalized data coded by the humans, and tested on data not used in training or tuning the models, the networks significantly improved the robot’s automatic estimation of the child’s behavior for most of the children in the study, beyond what would be estimated if the network combined all the children’s data in a “one-size-fits-all” approach, the researchers found. Rudovic and colleagues were also able to probe how the deep learning network made its estimations, which uncovered some interesting cultural differences between the children. “For instance, children from Japan showed more body movements during episodes of high engagement, while in Serbs large body movements were associated with disengagement episodes,” Rudovic notes.

Spotting fake images with AI

This tampered image (left) can be detected by noting visual artifacts (red rectangle, showing the unnaturally high contrast along the baseball player’s edges), compared to authentic regions (the parking lot background); and by noting noise pattern inconsistencies between the tampered regions and the background (as seen in “Noise” image). The “ground-truth” image is the outline of the added (fake) image used in the experiment. (credit: Adobe)

Thanks to user-friendly image editing software like Adobe Photoshop, it’s becoming increasingly difficult and time-consuming to spot some deceptive image manipulations.

Now, funded by DARPA, researchers at Adobe and the University of Maryland, College Park have turned to AI to detect the more subtle methods now used in doctoring images.

Forensic AI

What used to take an image-forensic expert several hours to do can now be done in seconds with AI, says Vlad Morariu, PhD, a senior research scientist at Adobe. “Using tens of thousands of examples of known, manipulated images, we successfully trained a deep learning neural network* to recognize image manipulation in each image,” he explains.

“We focused on three common tampering techniques — splicing, where parts of two different images are combined; copy-move, where objects in a photograph are moved or cloned from one place to another; and removal, where an object is removed from a photograph, and filled-in,” he notes.

The neural network looks for two things: changes to the red, green and blue color values of pixels; and inconsistencies in the random variations of color and brightness generated by a camera’s sensor or by later software manipulations, such as Gaussian smoothing.

Ref: 2018 Computer Vision and Pattern Recognition Proceedings (open access). Source: Adobe Blog.

* The researchers used a “two-stream Faster R-CNN” (a type of convolutional neural network) that they trained end-to-end to detect the tampered regions in a manipulated image. The two streams are RGB (red-green-blue — the millions of different colors) to find tampering artifacts like strong contrast difference and unnatural tampered boundaries; and noise (inconsistency of noise patterns between authentic and tampered regions — in the example above, note that the baseball player’s image is lighter, for example, in addition to more-subtle differences that can be detected by the algorithm — even what tampering technique was used). These two features are then fused together to further identify spatial co-occurrence of these two modalities (RGB and noise).