Removing empty lines from text/csv file with awk in linux

Problem

You have a text file that has empty lines, and you would like to remove them.

Solution

You can use awk in linux like the following to create a new file with the empty lines removed:

Comparing data from two big MySQL tables

Problem
You have two big tables in MySQL (>640K records), that maybe differ in the number of fields, but you want to make sure that the data in the common fields in both tables are the same.

Solution

  1. Use mysql to export the data from the first table in a csv file, selecting only the common fields.
    We use the /tmp folder on the server to make sure we have the right permissions to create the file:
  2. If the tables are in different databases remember to switch db:

    Export the second table in the second file:
  3. now use the diff. You can use any of the following options:
    • diff -q first_table.txt second_table.txt
    • diff first_table.txt second_table.txt > diff.txt