How to use Canvas to manipulate pixels

Modern browsers support video playback via the <video> element. Most browsers can also access the camera through the MediaDevices.getUserMedia() API. But even if these two things are combined, we cannot directly access and manipulate these pixels.

Fortunately, browsers have a Canvas API that allows us to draw graphics using JavaScript. We can actually draw images to <canvas> from the video itself, which allows us to manipulate and display those pixels.

What you learn here about how to manipulate pixels will provide you with the foundation for working with images and videos of any kind or source, not just canvas.

Before we start the video, let's see how to add an image to the canvas.

 <img src><div> <canvas id=Canvas class=video></canvas></div>

We create an image element to represent the image to be drawn on the canvas. Alternatively, we can use the Image object in JavaScript.

 var canvas;var context;function init() { var image = document.getElementById('SourceImage'); canvas = document.getElementById('Canvas'); context = canvas.getContext('2d'); drawImage(image); // Or // var image = new Image(); // image.onload = function () { // drawImage(image); // } // image.src = 'image.jpg';}function drawImage(image) { // Set the canvas the same width and height of the image canvas.width = image.width; canvas.height = image.height; context.drawImage(image, 0, 0);}window.addEventListener('load', init);

The above code draws the entire image to the canvas.

Check out the Paint image on canvas image by Welling Guzman (@wellingguzman) on CodePen.

Now we can start playing with those pixels!

Image data on the canvas allows us to manipulate and change pixels.

The data attribute is an ImageData object which has three properties - width, height and data/all of which represent something based on the original image. All these properties are read-only. What we care about is the data, a one-dimensional array represented by a Uint8ClampedArray object containing data for each pixel in RGBA format.

Even though a data property is read-only, it doesn't mean we can't change its value. This means we cannot assign another array to this property.

 // Get the canvas image datavar imageData = context.getImageData(0, 0, canvas.width, canvas.height);image.data = new Uint8ClampedArray(); // WRONGimage.data[1] = 0; // CORRECT

You may ask, what value does a Uint8ClampedArray object represent? Here's the description from MDN:

The Uint8ClampedArray type array represents an array of 8-bit unsigned integers that are clamped to 0-255; if you specify a value outside the range [0,255], 0 or 255 will be set; if you specify a non-integer, the nearest Integer will be set. The contents are initialized to 0. Once established, elements in the array can be referenced using the object's methods, or using standard array indexing syntax (i.e. using bracket notation)

In short, this array stores values ​​ranging from 0 to 255 at each location, which makes it a perfect solution for the RGBA format since each part is represented by a value from 0 to 255.

Color can be represented in RGBA format, which is a combination of red, green and blue. A represents the alpha value of the color's opacity.

Each position in the array represents a color (pixel) channel value.

• The first position is the red value
• The second position is the green value
• The third position is the blue value
• The fourth position is the Alpha value
• The 5th position is the next pixel red value
• The 6th position is the green value of the next pixel
• Position 7 is the blue value of the next pixel
• The 8th position is the next pixel Alpha value
• etc...

If you have a 2x2 image, then we have a 16 bit array (2x2 pixels x 4 values ​​each).

2x2 image reduced

The array will look like this:

 // RED GREEN BLUE WHITE[255, 0, 0, 255, 0, 255, 0, 255, 0, 0, 255, 255, 255, 255, 255, 255]

One of the quickest things we can do is set all pixels to white by changing all RGBA values ​​to 255.

 // Use a button to trigger the effectvar button = document.getElementById('Button');button.addEventListener('click', onClick);function changeToWhite(data) { for (var i = 0; i < data.length; i++) { data[i] = 255; }}function onClick() { var imageData = context.getImageData(0, 0, canvas.width, canvas.height); changeToWhite(imageData.data); // Update the canvas with the new data context.putImageData(imageData, 0, 0);}

The data will be passed as a reference, which means any modification we make to it, it will change the value of the passed parameter.

A nice effect that doesn't require too much calculation is to invert the colors of an image.

Color values ​​can be inverted using the XOR operator (^) or this formula 255 - value (value must be between 0-255).

 function invertColors(data) { for (var i = 0; i < data.length; i+= 4) { data[i] = data[i] ^ 255; // Invert Red data[i+1] = data[i +1] ^ 255; // Invert Green data[i+2] = data[i+2] ^ 255; // Invert Blue }}function onClick() { var imageData = context.getImageData(0, 0, canvas.width, canvas.height); invertColors(imageData.data); // Update the canvas with the new data context.putImageData(imageData, 0, 0);}

We are incrementing the loop by 4 instead of 1 like we did before, so we can fill 4 elements in the array from pixel to pixel, each pixel.

The alpha value has no effect on inverting the colors, so we skip it.

The brightness of an image can be adjusted using the following formula: newValue = currentValue + 255 * (brightness / 100).

1. Brightness must be between -100 and 100
2. currentValue is the current lighting value for red, green or blue.
3. newValue is the result of the current color light plus brightness
4. Adjusting the contrast of an image can be done using this formula:

 factor = (259 * (contrast + 255)) / (255 * (259 - contrast))color = GetPixelColor(x, y)newRed = Truncate(factor * (Red(color) - 128) + 128)newGreen = Truncate( factor * (Green(color) - 128) + 128)newBlue = Truncate(factor * (Blue(color) - 128) + 128)

The main calculation is to get the contrast factor that will be applied to each color value. Truncation is a function that ensures the value remains between 0 and 255.

Let's write these functions into JavaScript:

 function applyBrightness(data, brightness) { for (var i = 0; i < data.length; i+= 4) { data[i] += 255 * (brightness / 100); data[i+1] += 255 * (brightness / 100); data[i+2] += 255 * (brightness / 100); }}function truncateColor(value) { if (value < 0) { value = 0; } else if (value > 255) { value = 255; } return value;}function applyContrast(data, contrast) { var factor = (259.0 * (contrast + 255.0)) / ( 255.0 * (259.0 - contrast)); for (var i = 0; i < data.length; i+= 4) { data[i] = truncateColor(factor * (data[i] - 128.0) + 128.0); data[i+1] = truncateColor(factor * (data[i+1] - 128.0) + 128.0); data[i+2] = truncateColor( factor * (data[i+2] - 128.0) + 128.0); }}

In this case you don't need the truncateColor function because Uint8ClampedArray truncates the values, but to translate the algorithm we added in it.

One thing to remember is that if you apply brightness or contrast, the image data is overwritten and cannot be returned to its previous state. If we want to reset to the original state, the original image data must be stored separately for reference. Keeping the image variable accessible to other functions will be helpful because you can use the image to redraw the canvas and the original image.

 var image = document.getElementById('SourceImage'); function redrawImage() { context.drawImage(image, 0, 0);}

To make it work for video, we'll take our initial image script and HTML code and make some minor modifications.

Change the Image element of the video element by replacing the following line:

 <img src>

...with this:

 <video src></video>

JavaScript

Replace this line:

 var image = document.getElementById('SourceImage');

...add this line:

 var video = document.getElementById('SourceVideo');

To start processing the video, we have to wait until the video is ready to play.

 video.addEventListener('canplay', function () { // Set the canvas the same width and height of the video canvas.width = video.videoWidth; canvas.height = video.videoHeight; // Play the video video.play( ); // start drawing the frames drawFrame(video);});

Event playback is played for at least a few frames when there is enough data to play the media.

We cannot see any of the video displayed on the canvas because we are only displaying the first frame. We have to execute drawFrame every n milliseconds to keep up with the video frame rate.

Inside drawFrame, we call drawFrame again every 10ms.

 function drawFrame(video) { context.drawImage(video, 0, 0); setTimeout(function () { drawFrame(video); }, 10);}

After executing drawFrame, we create a loop that executes drawFrame every 10ms - enough time for the video to stay in sync within the canvas.

We can use the same function we created earlier to invert the colors:

 function invertColors(data) { for (var i = 0; i < data.length; i+= 4) { data[i] = data[i] ^ 255; // Invert Red data[i+1] = data[i +1] ^ 255; // Invert Green data[i+2] = data[i+2] ^ 255; // Invert Blue }}

and add this to drawFrame function:

 function drawFrame(video) { context.drawImage(video, 0, 0); var imageData = context.getImageData(0, 0, canvas.width, canvas.height); invertColors(imageData.data); context.putImageData(imageData, 0, 0); setTimeout(function () { drawFrame(video); }, 10);}

We can add a button and toggle the effect:

 function drawFrame(video) { context.drawImage(video, 0, 0); if (applyEffect) { var imageData = context.getImageData(0, 0, canvas.width, canvas.height); invertColors(imageData.data); context .putImageData(imageData, 0, 0); } setTimeout(function () { drawFrame(video); }, 10);}

We will keep the same code we used for the video, the only difference is that we will use MediaDevices.getUserMedia to change the video stream from a file to a camera stream.

MediaDevices.getUserMedia is a new API deprecating the previous API MediaDevices.getUserMedia(). Browsers still support older versions, and some browsers don't support newer versions, and we have to resort to polyfills to make sure the browser supports one of them.

First, remove the src attribute from the video element:

 <video><code></pre><pre><code>// Set the source of the video to the camera streamfunction initCamera(stream) { video.src = window.URL.createObjectURL(stream);}if (navigator .mediaDevices.getUserMedia) { navigator.mediaDevices.getUserMedia({video: true, audio: false}) .then(initCamera) .catch(console.error) );}

Live Demo

Everything we have covered so far is the foundation we need in order to create different effects for videos or images. We can use a lot of different effects by converting each color independently.

Converting colors to grayscale can be done in different ways using different formulas/techniques, to avoid getting too deep into the problem I'm going to show you five formulas based on the GIMP desaturate tool and Luma:

 Gray = 0.21R + 0.72G + 0.07B // LuminosityGray = (R + G + B) ÷ 3 // Average BrightnessGray = 0.299R + 0.587G + 0.114B // rec601 standardGray = 0.2126R + 0.7152G + 0.0722B / /ITU-RBT.709 standardGray = 0.2627R + 0.6780G + 0.0593B // ITU-R BT.2100 standard

What we want to find using these formulas is the brightness level of each pixel color. The value ranges from 0 (black) to 255 (white). These values ​​will create a grayscale (black and white) effect.

This means that the lightest color will be closest to 255 and the darkest color will be closest to 0.

Live Demo

The difference between a duotone effect and a grayscale effect is that two colors are used. In grayscale you have a gradient from black to white, whereas in duotone you have a gradient from any color to any other color (from blue to pink).

Using the intensity value of grayscale, we can replace it with the gradient value.

We need to create a gradient from ColorA to ColorB.

 function createGradient(colorA, colorB) { // Values ​​of the gradient from colorA to colorB var gradient = []; // the maximum color value is 255 var maxValue = 255; // Convert the hex color values ​​to RGB object var from = getRGBColor(colorA); var to = getRGBColor(colorB); // Creates 256 colors from Color A to Color B for (var i = 0; i <= maxValue; i++) { // IntensityB will go from 0 to 255 // IntensityA will go from 255 to 0 // IntensityA will decrease intensity while intensityB will increase // What this means is that ColorA will start solid and slowly transform into ColorB // If you look at it in other way the transparency of color A will increase and the transparency of color B will decrease var intensityB = i; var intensityA = maxValue - intensityB; // The formula below combines the two colors based on their intensity // (IntensityA * ColorA + IntensityB * ColorB) / maxValue gradient[i] = { r: (intensityA*from.r + intensityB*to.r) / maxValue, g: ( intensityA*from.g + intensityB*to.g) / maxValue, b: (intensityA*from.b + intensityB*to.b) / maxValue }; } return gradient;}// Helper function to convert 6digit hex values ​​to a RGB color objectfunction getRGBColor(hex){ var colorValue; if (hex[0] === '#') { hex = hex.substr(1); } colorValue = parseInt(hex, 16); return { r: colorValue >> 16, g: (colorValue >> 8) & 255, b: colorValue & 255 }}

In short, we create a set of color values ​​starting from color A, decreasing the intensity, while going to color B and increasing the intensity.

From #0096ff to #ff00f0

 var gradients = [ {r: 32, g: 144, b: 254}, {r: 41, g: 125, b: 253}, {r: 65, g: 112, b: 251}, {r: 91 , g: 96, b: 250}, {r: 118, g: 81, b: 248}, {r: 145, g: 65, b: 246}, {r: 172, g: 49, b: 245}, {r: 197, g: 34, b: 244}, {r: 220, g: 21 , b: 242}, {r: 241, g: 22, b: 242},];

Representation of scaling color transitions

Above is an example of a gradient of 10 color values ​​from #0096ff to #ff00f0.

Grayscale representation of color transitions

Now that we have a grayscale representation of the image, we can use it to map it to a duotone gradient value.

The duotone gradient has 256 colors while the grayscale has also 256 colors ranging from black (0) to white (255). That means a grayscale color value will map to a gradient element index.

 var gradientColors = createGradient('#0096ff', '#ff00f0');var imageData = context.getImageData(0, 0, canvas.width, canvas.height);applyGradient(imageData.data);for (var i = 0; i < data.length; i += 4) { // Get the each channel color value var redValue = data[i]; var greenValue = data[i+1]; var blueValue = data[i+2]; // Mapping the color values ​​to the gradient index // Replacing the grayscale color value with a color for the duotone gradient data[i] = gradientColors[redValue] .r; data[i+1] = gradientColors[greenValue].g; data[i+2] = gradientColors[blueValue].b; data[i+3] = 255;}

Live Demo

This topic could be more in-depth or explain more implications. The homework for you is to find different algorithms that can be applied to these skeleton examples.

Understanding the structure of pixels on the canvas will allow you to create an unlimited number of effects such as sepia, color blending, green screen effects, image flashes/glitches, etc.

You can even create effects on the fly without using images or videos

The above is the entire content of this article. I hope it will be helpful to everyone’s study. I also hope everyone will support VeVb Wulin Network.