crc32 vs crc64 | Senior Full Stack Developer, Coder, C#, Java, Linux, Network, PHP, Cloud

What is CRC-64?

A cyclic redundancy check (CRC) is an error-detecting code used to detect data corruption. When sending data, short checksum is generated based on data content and sent along with data. When receiving data, checksum is generated again and compared with sent checksum. If the two are equal, then there is no data corruption. The CRC-64 algorithm itself converts a variable-length string into an 16-character string.

It's not a research topic. It's really well understood: 
http://en.wikipedia.org/wiki/Cyclic_redundancy_check

The math is pretty simple. An 8-bit CRC boils all 
messages down to one of 256 values. If your message is 
more than a few bytes long, the possibility of multiple 
messages having the same hash value goes up higher and higher.

A 16-bit CRC, similarly, gives you one of the 65,536 
available hash values. What are the odds of any two 
messages having one of these values?

A 32-bit CRC gives you about 4 billion available hash values.

From the wikipedia article: "maximal total blocklength
 is equal to 2**r − 1". That's in bits. You don't need 
to do much research to see that 2**9 - 1 is 511 bits. 
Using CRC-8, multiple messages longer than 64 bytes will
 have the same CRC checksum value.

It's not a research topic. It's really well understood:

http://en.wikipedia.org/wiki/Cyclic_redundancy_check

The math is pretty simple. An 8-bit CRC boils all

messages down to one of 256 values. If your message is

more than a few bytes long, the possibility of multiple

messages having the same hash value goes up higher and higher.

A 16-bit CRC, similarly, gives you one of the 65,536

available hash values. What are the odds of any two

messages having one of these values?

A 32-bit CRC gives you about 4 billion available hash values.

From the wikipedia article: "maximal total blocklength

is equal to 2**r − 1". That's in bits. You don't need

to do much research to see that 2**9 - 1 is 511 bits.

Using CRC-8, multiple messages longer than 64 bytes will

have the same CRC checksum value.

The effectiveness of a CRC is dependent on multiple factors. 
You not only need to select the SIZE of the CRC but also the 
GENERATING POLYNOMIAL to use. There are complicated and 
non-intuitive trade-offs depending on:

The expected bit error rate of the channel.
Whether the errors tend to occur in bursts or tend to 
be spread out (burst is common)
The length of the data to be protected - maximum length,
 minimum length and distribution.
The paper Cyclic Redundancy Code Polynominal Selection 
For Embedded Networks, by Philip Koopman and Tridib Chakravarty, 
publised in the proceedings of the 2004 International Conference
 on Dependable Systems and Networks gives a very good overview 
and makes several recomendations. It also provides a bibliography 
for further understanding.

http://www.ece.cmu.edu/~koopman/roses/dsn04/koopman04_crc_poly_embedded.pdf

The effectiveness of a CRC is dependent on multiple factors.

You not only need to select the SIZE of the CRC but also the

GENERATING POLYNOMIAL to use. There are complicated and

non-intuitive trade-offs depending on:

The expected bit error rate of the channel.

Whether the errors tend to occur in bursts or tend to

be spread out (burst is common)

The length of the data to be protected - maximum length,

minimum length and distribution.

The paper Cyclic Redundancy Code Polynominal Selection

For Embedded Networks, by Philip Koopman and Tridib Chakravarty,

publised in the proceedings of the 2004 International Conference

on Dependable Systems and Networks gives a very good overview

and makes several recomendations. It also provides a bibliography

for further understanding.

http://www.ece.cmu.edu/~koopman/roses/dsn04/koopman04_crc_poly_embedded.pdf

File : PDF : Click : Cyclic-Redundancy-Code-CRC-Polynomial-Selection-For-Embedded-Networks_koopman04_crc_poly_embedded

File : sha1_comparison_Which_is-a-more-accurate-method-of-duplicate-file-detection_ SHA1 or CRC32_ – Quora

File:CRC Data Sets Tests : PDF-DOWNLOAD_kutayzorlu.com_18.2_million_dataset_CRC-64_test_Program_source

File : Amazon Red Shift : Pdf Download : kutayzorlu.com_amazon_red_redshift-dg

The actual playable content (audio, video, timed text, 
clips, and artwork) is stored in several places. Original 
source material and the encoded copies are stored on Amazon S3.

If the title is currently available for viewing,
 a copy of the audio, video, timed text, and clips are 
put on one or more Netflix Open Connect appliances. 
Users stream movies and shows off of the appliances.

A copy of static assets such as thumbnails and box-shots 
are stored on commercial Content Delivery Networks. 
The exact location and number of copies is managed by the CDN.

The actual playable content (audio, video, timed text,

clips, and artwork) is stored in several places. Original

source material and the encoded copies are stored on Amazon S3.

If the title is currently available for viewing,

a copy of the audio, video, timed text, and clips are

put on one or more Netflix Open Connect appliances.

Users stream movies and shows off of the appliances.

A copy of static assets such as thumbnails and box-shots

are stored on commercial Content Delivery Networks.

The exact location and number of copies is managed by the CDN.

SHA1- JAVA checksum verify

package main;
 
import java.io.FileInputStream;
import java.io.IOException;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
 

public class HashFileTest {
     
    public static void main(String[] args) throws NoSuchAlgorithmException, IOException {
        boolean result = verifyChecksum("/home/eclipse-jee-indigo-SR2-linux-gtk-x86_64.tar.gz", "177750b65a21a9043105fd0820b85b58cf148ae4");
        System.out.println("Does the file's checksum matches the expected one? " + result);
    }
     
    /**
     * Verifies file's SHA1 checksum
     * @param Filepath and name of a file that is to be verified
     * @param testChecksum the expected checksum
     * @return true if the expeceted SHA1 checksum matches the file's SHA1 checksum; false otherwise.
     * @throws NoSuchAlgorithmException
     * @throws IOException
     */
    public static boolean verifyChecksum(String file, String testChecksum) throws NoSuchAlgorithmException, IOException
    {
        MessageDigest sha1 = MessageDigest.getInstance("SHA1");
        FileInputStream fis = new FileInputStream(file);
  
        byte[] data = new byte[1024];
        int read = 0; 
        while ((read = fis.read(data)) != -1) {
            sha1.update(data, 0, read);
        };
        byte[] hashBytes = sha1.digest();
  
        StringBuffer sb = new StringBuffer();
        for (int i = 0; i < hashBytes.length; i++) {
          sb.append(Integer.toString((hashBytes[i] & 0xff) + 0x100, 16).substring(1));
        }
         
        String fileHash = sb.toString();
         
        return fileHash.equals(testChecksum);
    }
 
 
}

package main;

import java.io.FileInputStream;

import java.io.IOException;

import java.security.MessageDigest;

import java.security.NoSuchAlgorithmException;

public class HashFileTest {

public static void main(String[] args) throws NoSuchAlgorithmException, IOException {

boolean result = verifyChecksum("/home/eclipse-jee-indigo-SR2-linux-gtk-x86_64.tar.gz", "177750b65a21a9043105fd0820b85b58cf148ae4");

System.out.println("Does the file's checksum matches the expected one? " + result);

}

/**

* Verifies file's SHA1 checksum

* @param Filepath and name of a file that is to be verified

* @param testChecksum the expected checksum

* @return true if the expeceted SHA1 checksum matches the file's SHA1 checksum; false otherwise.

* @throws NoSuchAlgorithmException

* @throws IOException

public static boolean verifyChecksum(String file, String testChecksum) throws NoSuchAlgorithmException, IOException

{

MessageDigest sha1 = MessageDigest.getInstance("SHA1");

FileInputStream fis = new FileInputStream(file);

byte[] data = new byte[1024];

int read = 0;

while ((read = fis.read(data)) != -1) {

sha1.update(data, 0, read);

};

byte[] hashBytes = sha1.digest();

StringBuffer sb = new StringBuffer();

for (int i = 0; i < hashBytes.length; i++) {

sb.append(Integer.toString((hashBytes[i] & 0xff) + 0x100, 16).substring(1));

}

String fileHash = sb.toString();

return fileHash.equals(testChecksum);

}

SHA1- TEXT String

package main;
 
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
 
public class HashTextTest {
 
    /**
     * @param args
     * @throws NoSuchAlgorithmException 
     */
    public static void main(String[] args) throws NoSuchAlgorithmException {
        System.out.println(sha1("test string to sha1"));
    }
     
    static String sha1(String input) throws NoSuchAlgorithmException {
        MessageDigest mDigest = MessageDigest.getInstance("SHA1");
        byte[] result = mDigest.digest(input.getBytes());
        StringBuffer sb = new StringBuffer();
        for (int i = 0; i < result.length; i++) {
            sb.append(Integer.toString((result[i] & 0xff) + 0x100, 16).substring(1));
        }
         
        return sb.toString();
    }
}

package main;

import java.security.MessageDigest;

import java.security.NoSuchAlgorithmException;

public class HashTextTest {

/**

* @param args

* @throws NoSuchAlgorithmException

public static void main(String[] args) throws NoSuchAlgorithmException {

System.out.println(sha1("test string to sha1"));

}

static String sha1(String input) throws NoSuchAlgorithmException {

MessageDigest mDigest = MessageDigest.getInstance("SHA1");

byte[] result = mDigest.digest(input.getBytes());

StringBuffer sb = new StringBuffer();

for (int i = 0; i < result.length; i++) {

sb.append(Integer.toString((result[i] & 0xff) + 0x100, 16).substring(1));

}

return sb.toString();

}

SHA1 – File – JAVA

  public static String sha1(final File file) throws NoSuchAlgorithmException, IOException {
    final MessageDigest messageDigest = MessageDigest.getInstance("SHA1");

    try (InputStream is = new BufferedInputStream(new FileInputStream(file))) {
      final byte[] buffer = new byte[1024];
      for (int read = 0; (read = is.read(buffer)) != -1;) {
        messageDigest.update(buffer, 0, read);
      }
    }

    // Convert the byte to hex format
    try (Formatter formatter = new Formatter()) {
      for (final byte b : messageDigest.digest()) {
        formatter.format("%02x", b);
      }
      return formatter.toString();
    }
  }

public static String sha1(final File file) throws NoSuchAlgorithmException, IOException {

final MessageDigest messageDigest = MessageDigest.getInstance("SHA1");

try (InputStream is = new BufferedInputStream(new FileInputStream(file))) {

final byte[] buffer = new byte[1024];

for (int read = 0; (read = is.read(buffer)) != -1;) {

messageDigest.update(buffer, 0, read);

}

// Convert the byte to hex format

try (Formatter formatter = new Formatter()) {

for (final byte b : messageDigest.digest()) {

formatter.format("%02x", b);

}

return formatter.toString();

}

public byte[] createSha1(File file) throws Exception  {
    MessageDigest digest = MessageDigest.getInstance("SHA-1");
    InputStream fis = new FileInputStream(file);
    int n = 0;
    byte[] buffer = new byte[8192];
    while (n != -1) {
        n = fis.read(buffer);
        if (n > 0) {
            digest.update(buffer, 0, n);
        }
    }
    return digest.digest();
}

public byte[] createSha1(File file) throws Exception {

MessageDigest digest = MessageDigest.getInstance("SHA-1");

InputStream fis = new FileInputStream(file);

int n = 0;

byte[] buffer = new byte[8192];

while (n != -1) {

n = fis.read(buffer);

if (n > 0) {

digest.update(buffer, 0, n);

}

return digest.digest();

}

References

https://crc64.online/
http://www.unit-conversion.info/texttools/crc/
https://www.nitrxgen.net/hashgen/
http://www.sha1-online.com/
https://toolslick.com/programming/hashing/crc-calculator#
https://en.wikipedia.org/wiki/Cyclic_redundancy_check
https://searchcode.com/codesearch/view/22078345/
https://jar-download.com/artifacts/com.github.tonivade/claudb/1.0.2/source-code/com/github/tonivade/claudb/persistence/CRC64.java
https://stackoverflow.com/questions/2321676/data-length-vs-crc-length
https://stackoverflow.com/questions/20562546/how-to-get-crc64-distributed-calculation-use-its-linearity-property
http://www.sha1-online.com/sha1-java/