How to datamosh videos

Datamoshing is the process of intentionally damaging media files with the goal of creating interesting distortion effects during playback. In some cases the term datamoshing is used to describe this process applied to any type of media file — I like to think it applies solely to video since it results in moving images being moshed together. Regardless of the application of the term, datamoshing videos can be done quite easily with free, cross-platform tools.

Modern compressed video files have very complex methods of reducing the amount of storage or bandwidth needed to display the video. To do this most formats don’t store the entire image for each frame.

Frames which store an entire picture are called I-frames (Intra-coded), and can be displayed without any additional information.

Frames which don’t contain the entire picture require information from other frames in order to be displayed, either previous or subsequent frames, these frames are called P-frames (Predicted) and B-frames (Bi-predictive). Instead of storing full pictures these P-frames and B-frames contain data describing only the differences in the picture from the preceding frame, and/or from the next frame, this data is much smaller compared to storing the entire picture — especially in videos where there isn’t much movement.

When a video is encoded, or compressed, a combination of these types of frames are used. In most cases this means many P-frames with I-frames interspersed at regular intervals and where drastic visual changes in the video occur. More information on frame types can be found here.

If an I-frame is corrupted, removed or replaced the data contained in the following P-frames is applied to the wrong picture. In the above video I-frames have been removed and so instead of scenes changing properly you see the motion from a new scene applied to a picture from a previous frame. This process of corrupting, removing or replacing I-frames is a very popular video datamoshing technique and what this tutorial will focus on.

Another video datamoshing technique involves selecting one or more P-frames and duplicating them multiple times consecutively. This results in the same P-frame data being applied to one picture over and over again, accentuating the movement and creating what’s known as a Bloom effect.

For this tutorial we’ll be using Avidemux, a free, cross platform video editing application. Generally the effects of datamoshing are viewed as errors, or undesirable and thus applications like Avidemux try their best to correct these errors and eliminate glitching distortion. For this reason the latest version of Avidemux isn’t very good for datamoshing, but some older versions, such as 2.5.6, available here, work just fine.

After downloading and installing Avidemux 2.5.6 Open the video you want to mosh.

avidemux-open

Avidemux may show warnings depending on the type of file you’re using, select No and continue.

avidemux-h.264-detected

Once the video is loaded we’ll be making a small change to allow us to remove I-frames and still have a playable video. Under Video on the left side of the interface use the dropdown to change the selection from Copy to MPEG-4 ASP (Xvid).

Avidemux-Video

Next click the Configure button below the Video dropdown on the left. Select the Frame tab and then change the Maximum I-frame Interval from 300 to 99999999 then click OK. By changing this setting we’re allowing the video file to be played back even if it has unusually few I-frames, one every 99999999 frames.

Maximum-I-Frame-Interval

With this setting changed the video must be saved and then reloaded for it to take effect. Save the video with a new name to indicate that the Maximum I-frame Interval has been adjusted.

Avidemux-Save

Open the new video, select No if Avidemux displays any warnings. Once opened, change the Video dropdown on the left side of the interface back to Copy. We won’t be changing the encoding or any settings of the video at this point, we’re just going to remove I-frames and then Save it.

Avidemux-copy

To remove I-frames we use the slider at the bottom of the interface, it displays the current frame type below the video navigation buttons, you’ll mostly see Frame Type: I (00) and Frame Type: P (00). The first frame will most likely be an I-frame and should be left in so that the video can start properly. To locate other I-frames click on the slider tab/grip to focus the slider then press the Up Arrow on your keyboard to jump to the next I-frame, the Down Arrow will jump to the previous I-frame.

In order to remove an I-frame we must select it, this is done by marking an in point and an out point, these points are referred to as A and B in Avidemux and the frames of video between these two points are considered selected. Once you have found an I-frame click on the mark A button under the slider. You should see a blue border identify the new selection, starting at the slider grip and encompassing the remaining frames in the video.

Avidemux-A

Pressing delete now would delete the current I-frame and all subsequent frames so we have to reduce the selection to only the I-frame. This is done be pressing the Right Arrow key to move to the next frame and then clicking the mark B button below the slider. The blue selection border should update to show only the I-frame selected as illustrated below.

Avidemux-B

Now that only a single I-frame is selected press the Delete key on your keyboard to remove it. To remove all the I-frames use the Up Arrow to move to the next one and repeat the removal process. Try to avoid moving backwards through the video once you’ve removed I-frames as this can cause Avidemux to crash, stick to moving forward through the I-frames and removing them.

After you’ve removed one or more I-frames, Save the video and again select No if Avidemux prompts or warns you about something, smart copy for instance.

Avidemux-Enable-Smartcopy

Once the video is saved open it up in your favorite player and evaluate the havoc you’ve wrought upon it.

The video included in this post was datamoshed using this technique, however the audio was slowed down using traditional video editing.#corruptabsolutely

Windows

OSX

How to glitch images using RGB channel shifting

Channel shifting is the process of taking the red, green or blue values of pixels in an image and applying those values to pixels in different positions on the image. In this tutorial we are going to accomplish this effect using the Processing language.

If you don’t have the time or inclination to glitch images using scripts you can use dedicated apps such as Glitch for iOS.

glitch4ios

To get started download and install the latest version of Processing, version 3.1.1 at the time of writing this. I’ve written a channel shifting script you can download here, you’ll have to unzip it once it’s downloaded. Once you’ve installed and opened Processing you can load the script by accessing the menu.

File > Open

And navigating to the ChannelShiftGlitch.pde script file. In the script, which are referred to as sketches in Processing, you’ll need to change the following lines to point the script at the image you want to glitch. It’s easiest to place the image in the same directory as the script file.

// image path is relative to sketch directory
String imgFileName = "MyImage";
String fileType = "jpg";

I’ve set up some settings variables to make the script easier to use, you can see these towards the top of the script under the comment of script settings.

// repeat the process this many times
int iterations = 5;

// use result image as new source for iterations
boolean recursiveIterations = false;

// shift the image vertically true/false
boolean shiftVertically = false;

// shift the image horizontally true/false
boolean shiftHorizontally = true;

This script is able to apply the same channel shifting effect multiple times, the number of times is specified by the iterations variable, currently set to 5. This variable drives a for loop around the channel shifting code as seen below.

// repeat the process according 
// to the iterations variable
    for(int i = 0;i < iterations;i++)
    {
      // generate random numbers 
      // for which channels to swap
      int sourceChannel = int(random(3));
      int targetChannel = int(random(3));

You can also see in this code where the script generates a random number which will determine which of the three channels, red, green or blue are used as a source, and which of the three channels are used as a target. Next the script sets up the shifting positions, how far vertically and how far horizontally the channel should be shifted. These are either 0 if shifting is set to false for that plane (determined by the shiftHorizontally and shiftVertically settings), or a random number between the 0 and the height or width of the image.

// start with no horizontalShift 
int horizontalShift = 0; 

// if shiftHorizontally is true generate a 
// random number to shift horizontally by
if(shiftHorizontally)
  horizontalShift = int(random(width));
      
// start with no verticalShift 
int verticalShift = 0;
      
// if shiftVertically is true generate a 
// random number to shift vertically by
if(shiftVertically)
  verticalShift = int(random(height));

Next the script calls the main copyChannel method. This method accepts pixel arrays of the source and target images and will copy one channel to another from one part of the image to another and wrap around both horizontally and vertically if it runs out of space.

// shift the channel
copyChannel(
  sourceImg.pixels, 
  targetImg.pixels, 
  verticalShift, 
  horizontalShift, 
  sourceChannel, 
  targetChannel
  );

The method starts by starting a counter to loop through the rows of pixels in the image, top-to-bottom. This counter is added to the sourceYOffset variable to apply the vertical shift. If the vertical shift plus the counter is higher than the image height we subtract the image height to wrap the shift around to the top of the image.

// starting at the sourceY and pointerY
// loop through the rows
for(int y = 0; y < height; y++) { 
  // add y counter to sourceY 
  int sourceYOffset = sourceY + y; 

  // wrap around the top of the 
  // image if we've hit the bottom 
  if(sourceYOffset >= height)
    sourceYOffset -= height;

Within the row loop the script starts another counter to loop through the columns in that row, left-to-right. It also adds that counter to the sourceXOffset to apply the horizontal shift. If the horizontal shift plus the counter is wider than the image width we subtract the image width to wrap the shift around to the left of the image.

// starting at the sourceX and pointerX 
// loop through the pixels in this row
for(int x = 0; x < width; x++) 
{ 
  // add x counter to sourceX 
  int sourceXOffset = sourceX + x; 

  // wrap around the left side of the 
  // image if we've hit the right side 
  if(sourceXOffset >= width)
    sourceXOffset -= width;

Processing stores image pixels in an array as illustrated in the image below.

pixelarray

In order to access a pixel at specific x/y coordinates in the image we use the formula below.

y * width + x

Next the script isolates the RGB (red, green, blue) values for both the source and target pixels by using the formula above to access the pixel and then some Processing methods to extract the separate RGB channel values.

// get the color of the source pixel
color sourcePixel = 
  sourcePixels[sourceYOffset * width + sourceXOffset];
            
// get the RGB values of the source pixel
float sourceRed = red(sourcePixel);
float sourceGreen = green(sourcePixel);
float sourceBlue = blue(sourcePixel);
   
// get the color of the target pixel
color targetPixel = targetPixels[y * width + x]; 

// get the RGB of the target pixel
// two of the RGB channel values are required 
// to create the new target color
// the new target color is two of the target
// RGB channel values and one RGB channel value 
// from the source
float targetRed = red(targetPixel);
float targetGreen = green(targetPixel);
float targetBlue = blue(targetPixel);

Now that the script has the RGB of the source pixel and RGB of the target pixel we can proceed to shift one of the channels. We use a switch statement for this, deciding which source channel to use based on the sourceChannel variable which has a holds a random number we generated earlier, either 0, 1 or 2.

// create a variable to hold 
// the new source RGB channel value
float sourceChannelValue = 0;
            
// assigned the source channel value 
// based on sourceChannel random number passed in
switch(sourceChannel)
{
  case 0:
    // use red channel from source
    sourceChannelValue = sourceRed;
    break;
  case 1:
    // use green channel from source
    sourceChannelValue = sourceGreen;
    break;
  case 2:
    // use blue channel from source
    sourceChannelValue = sourceBlue;
    break;
}

After selecting a source channel we apply that channel value to either the red, green or blue channel of the target pixel, again using a switch statement, this time based on the targetChannel variable.

// assigned the source channel value to a 
// target channel based on targetChannel 
// random number passed in
switch(targetChannel)
{
  case 0:
    // assign value to target red channel
    targetPixels[y * width + x] = 
      color(sourceChannelValue, 
        targetGreen, 
        targetBlue);
    break;
 case 1:
    // assign value to target green channel
    targetPixels[y * width + x] = 
      color(targetRed, 
        sourceChannelValue, 
        targetBlue);
    break;
 case 2:
    // assign value to target blue channel
    targetPixels[y * width + x] = 
      color(targetRed, 
        targetGreen, 
        sourceChannelValue);
    break;
}

That’s it for the copyChannel method. The channel has been shifted in the target image at this point. Back in the main draw method of the script there is an if statement that determines whether or not the next iteration (if the iterations variable is to to greater than 1) will use the original image as a source, or use the new shifted image as a source.

// use the target as the new source 
// for the next iteration
if(recursiveIterations)
  sourceImg.pixels = targetImg.pixels;

Using the original image as a source for more than 3 iterations is rather pointless because there are only three channels in the original image to shift to around, always resulting in three shifted ghost images. So if you set iterations higher than 3 you should probably set recursiveIterations to true.

Vertical-Channel-Shift

Setting the recursiveIterations variable to true at the beginning of the script will use each new shifted image as a source for the next iteration and will result in much more dynamic results when iterations is set higher than 3, say 25 or 50.

Recursive-Channel-Shift

Personally I prefer restricting the shifting to either horizontal or vertical alone, but the script allows for the combination  by changing the shiftVertically and shiftHorizontally settings. You can find more Processing tutorials here, and remember, if you’re going to corrupt, corrupt absolutely. #corruptabsolutely

iOS (iPhone, iPad)

Windows

OSX

Processing

How to glitch images with WordPad

Sometimes glitching images can be as easy as opening them and saving them again — and that’s exactly the case when image bending with Microsoft WordPad. WordPad is a basic word processor that is included with almost all versions of Microsoft Windows from Windows 95 onwards. If you are running Microsoft Windows then you’ve probably got WordPad on your system, and you can datamosh images with it.

If you don’t have the time or inclination to glitch images using WordPad you can use dedicated apps such as Glitch for iOS.

glitch4ios

To start glitching with WordPad we need to convert our source image to a BMP (Bitmap) file. This can be done with any image editing software, but since this tutorial is about WordPad we’re going to use one of its fellow Microsoft applications, Microsoft Paint, which is also included with most versions of Microsoft Windows. If you can’t find Paint, press the Win + R keys, type in “pbrush” and press Enter, Paint should open. Open your file and then choose Save as and BMP picture.

Paint-save-as

Then select 24-bit Bitmap (*.bmp;*.dib) from the Save as type dropdown, other types of BMP will work as well to varying degrees.

Paint-save-as-bitmap

Next we need to open the BMP in WordPad. If you can’t find WordPad, press the Win + R keys, type in “write” and press Enter, WordPad should open. Open your BMP image in WordPad by selecting All Documents(*.*) in the dropdown beside the File name field.

WordPad-open

Wait for WordPad to load the image as a document and then press Ctrl+S on your keyboard to Save or use the menu to Save the file.

That’s all there is to it. The process of opening and saving the image using WordPad invokes what has been dubbed the WordPad Effect. You can also try deleting some random text from the file before saving, but it’s not required to glitch the image. The splash image for this post was simply opened and then saved, the WordPad Effect did the rest.

WordPad-opened

So what’s going on here? For the curious, a cursory comparison of the before and after files seems to confirm my suspicion that the transformation, at least in some part, has to do with WordPad converting line breaks when it loads in the image. Line breaks are some bytes that tell text editors and word processors where they should break the text and continue on the next line.

Different operating systems handled them slightly differently, the bytes involved are what’s called a Line Feed, represented by 0x0A in hexadecimal, and a carriage return, 0x0D in hexadecimal. This nomenclature comes from mechanical typewriters where a line of physical paper is fed through the roller (line feed) and the carriage is returned to the beginning of the line (carriage return). In text editors the meaning is the same, a 0x0A tells the editor to move to the next line and a 0x0D tells the editor to move to the beginning of the line.

Image files don’t contain any line feeds or carriage returns, but since WordPad looks at the image data as if it were text it looks for 0x0D and 0x0A in the image file which mean something completely unrelated for images. WordPad attempts to clean up what it thinks is a text file by ensuring that all line breaks have a line feed and a carriage return, so when it finds 0x0D without an accompanying 0x0A, it adds one, and when it finds an 0x0A without an 0x0D it adds one there as well. Because 0x0A and 0x0D mean different things for images, when WordPad thinks it’s correcting a text file in actuality its corrupting the image — et voilà, the WordPad Effect. #corruptabsolutely

How to glitch images using Processing scripts

Datamoshing images, also known as databending or glitching images can be done in a number of ways, some of the most interesting glitches are accomplished by using the Processing programming language. From the beginning the Processing language, was designed as a first programming language. It was inspired by earlier languages like BASIC and Logo.

If you don’t have the time or inclination to glitch images using scripts you can use dedicated apps such as Glitch for iOS.

glitch4ios

To get started download and install the latest version of Processing, version 3.1.1 at the time of writing this. I’ve written a simple script you can download here, you’ll have to unzip it once it’s downloaded. Once you’ve installed and opened Processing you can load the script by accessing the menu.

File > Open

And navigating to the SimpleGlitch.pde script file. In the script, which are referred to as sketches in Processing, you’ll need to change the following lines to point the script at the image you want to pixel sort:

// image path is relative to sketch directory
PImage img;
String imgFileName = "MyImage";
String fileType = "jpg";

In the simple glitch sketch we’re doing a little more. For each pixel in the image the script generates some random numbers to determine whether or not to glitch that pixel. It also keeps track of whether or not the previous pixel was glitched by setting the previousPixelGlitched variable to true or false, if it was, there’s a higher chance that the code will glitch the current pixel. This type of structure will result in lines of glitched pixels, rather that just randomly positioned glitched pixels, which ends up looking like static.

The sketch generates a new random color for the randomColor variable before glitching any pixels and each time a pixel is not glitched. This means that each line of glitched pixels will have a new random color available to it.

// random color 
// 0-255, red, green, blue, alpha
color randomColor = color(random(255), random(255), random(255), 255);

The sketch generates another random number, this time between .5 and 1, and uses this as a mix ratio to mix the random color with the current pixel’s color.

// percentage to mix
float mixPercentage = .5 + random(50)/100;

// mix colors by random percentage of new random color
img.pixels[y + x * img.height] = lerpColor(pixelColor, randomColor, mixPercentage);

For the featured image of this post I adjusted the random color generator to always use 255 (the maximum) blue and thus the resulting image contains colors from the cool range of the spectrum.

In short, this script creates lines of random length and of random colors and mixes them into the original image. I also added some commented out lines that illustrate how to apply filters to the entire image in Processing, uncomment them to see how they affect the result.

// apply some filters
// https://processing.org/reference/filter_.html

// posterize filter
// filter(POSTERIZE, 4);

// dilate filter
// filter(DILATE);

Some ideas for experimenting with this script might be changing the mixPercentage randomness, or, as I did, adjust the random color to be less random by replacing any of the three random(255) with a number between 0 and 255. Instead of glitching pixels randomly you could use a counter, or geometric function (sin, cos, etc) in the loop to glitch pixels in mathematical patterns.

If you’re feeling intimidated by the simple glitch script I created a super simple glitch script you can download here. The super simple glitch script doesn’t actually do anything except loop through each pixel in the image, so it’s ready for experimentation. #corruptabsolutely

iOS (iPhone, iPad)

Windows

OSX

Processing

How to glitch images using audio editing software

Images can be pleasantly destroyed in a great number of ways, some of the best results come from applying transformation algorithms to the raw image data. Applying filter algorithms to images is something one would normally use software like Adobe Photoshop for, however, using audio processing software instead can yield much more interesting, and unexpected, results.

If you don’t have the time or inclination to manually glitch images with audio processing software you can use dedicated apps such as Glitch for iOS.

glitch4ios

Firstly we’ll need some audio processing software, many will work, Audacity is free, supports many platforms and works quite well for glitching images. Secondly we’ll need a databending-friendly image, the BMP format works well for this type of bending. Once we have an image and Audacity installed, open Audacity and import the image by selecting:

File > Import > Raw Data

audacity-importraw

Audacity will then ask for some information about the file we are importing, we’re going to lie, for Encoding select either U-Law or A-Law. We will have to export with the same encoding setting so remember which was selected. Defaults will work fine for the rest of the import options.

audacity-import

The image will now be open as an audio file, I don’t suggest pressing play. Now we can select any portion of the file or its entirety by clicking and dragging on the waveform (the chart-like display). In some cases it’s better avoid selecting the beginning (first 5-10 seconds of the waveform) of the file as this contains the file header, a section of the file which contains information needed to display the image, if the image won’t display after exporting consider leaving the header intact.

Once we have a selection, we can apply any of the filters under the Effects menu. I have found the Invert, Reverb, Reverse, Wahwah, Compressor and Echo work quite well, but here is where you can experiment. You’ll see the waveform change as each filter effect is applied. The hero image of this post was created using the Compressor and Echo filters applied to the entire file.

Once we’ve applied one or more filter effects we can export the data back to an image by selecting:

File > Export Audio

Change the filename back to the proper image extension, in this case BMP. The Save as type should be set to Other uncompressed files, the Header should be set to RAW (header-less) and the Encoding should be set to either U-Law or A-Law depending on which was chosen during the import process.

audacity-export

Audacity may complain that not all is well with the filename, and perhaps prompt for metadata but these prompts can be accepted and ignored.

All that’s left is to check the result — in an image viewer, not a music player. #corruptabsolutely

Audacity
Audacity Forums