Mount and sync a USB drive on RPi Ubuntu with GPIO

Actions:

1. So first the USB is plugged in.
2. It is automounted using a GPIO button press. Flash red if this fails.
   
3. Blue light will flash light during rsync cp (copy). If an error occurs show red LED and turn off blue LED.
4. Send email with results on completion.
5. Unmount USB and show steady blue LED indication turn off blue LED.

Ok lets do it

• For auto mounting, the package usbmount seems too problematic (won’t mount NTFS drives) after trying these instructions.
• it took much longer than I’d have liked to get to the point of using this mount command with sudo:

  mount -L SneakerNet -o uid=ubuntu,gid=deluge,dmask=007,fmask=117 /media/USB

This allows me to mount the USB by it’s label (SneakerNet) alone. This I think is ideal if I want to use a different USB drive.

• I can put the mount command in a script and chmod 750

  #!/bin/bash
  # mount USB drive
  
  mount -L SneakerNet -o uid=ubuntu,gid=deluge,dmask=007,fmask=117 /media/sneaker$

• The next problem is auto mounting a USB drive. In ubuntu server this is surprisingly difficult, this page shows how incomplete this is. udev while promising doesn’t work for NTFS (this came close), which took me a long time to realise – even invoking a python script from a bash script called by udev… pmount requires the device name, which can change depending on whats plugged in, usbmount doesn’t work, udisks2 is out (doesn’t allow the required options for mounting) :/
• this looks interesting
• for exfat, this could work

  sudo apt-get install exfat-utils exfat-fuse

• So I’m trying a python implementation of udev (sigh x2). Trying this example.
  • Install pyudev

    sudo pip install pyudev

• I think I’ll skip auto mounting and try mounting from a button press (in python as previously installed).

  #!/usr/bin/env python
  
  # mount USB drive
  
  import subprocess
  
  import time
  
  mountCommand = "mount -L SneakerNet -o uid=ubuntu,gid=deluge,dmask=002,fmask=113 /media/SneakerNet"
  
  time.sleep(1)
  
  process = subprocess.Popen(mountCommand.split(), stdout=subprocess.PIPE)
  
  output = process.communicate()[0]

  Well that little script took all of 5 minutes and negated 2 days work..

• Now I’ll add in rsync, logging and email.

  #!/usr/bin/env python
  import subprocess
  import os
  from email.mime.text import MIMEText
  
  mountCommand = ["mount", "-L", "SneakerNet", "-o", "uid=ubuntu,gid=deluge,dmask=002,fmask=113", "/media/SneakerNet"]
  umountCommand = ["umount", "/media/SneakerNet"]
  mkdirCommand = ["mkdir", "/media/SneakerNet/Done"]
  
  if os.path.ismount("/media/SneakerNet/") == 0:
      MountProcess = subprocess.Popen(mountCommand, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
      MountData, MountError  = MountProcess.communicate()
      if len(MountData) == 37:
          mountfail = 1
          logs = "No USB with label SneakerNet found."
      elif len(MountData) == 0:
          logs = "Mounted"
          mountfail = 0
      else:
          logs = "Coding error or no su maybe?)"
          mountfail = 1
  else:
      logs = "Already mounted"
      mountfail = -1
  
  rsyncCommand = ["rsync", "--delete", "-ahP", "--no-D", "--log-file=/media/SneakerNet/RsyncLog.txt", "/media/Data/Deluge/Done/", "/media/SneakerNet/Done"] #--dry-run #still have -P
  
  if mountfail < 1:
      if os.path.isdir("/media/SneakerNet/Done") == 0:
          subprocess.Popen(mkdirCommand)
          logs += "\nMade Directory /media/SneakerNet/Done"
      if os.path.isfile("/media/SneakerNet/RsyncLog.txt") == 1:
          with open("/media/SneakerNet/RsyncLogArch.txt", "a") as RsyncLogArch, open('/media/SneakerNet/RsyncLog.txt', 'r') as rsynclog:
              for line in rsynclog:
                  RsyncLogArch.write(line)
              logs += "\nMoved old rsync log to archive"
      with open('/media/SneakerNet/RsyncLog.txt', 'w') as rsynclog:
          rsynclog.write("Python output:\n")
          logs += "\nEmptied Log for rsync:\n"
          rsynclog.write(logs)
      print logs #for GUI...
      subprocess.call(rsyncCommand) ##### USING CALL (only for GUI)
  
      rsynclog = open('/media/SneakerNet/RsyncLog.txt', 'rb')
      msg = MIMEText(rsynclog.read())
      rsynclog.close()
      msg["From"] = "python@parko.xyz"
      msg["To"] = "phaseform@gmail.com"
      msg["Subject"] = "RPi rsync log"
      emailProcess = subprocess.Popen(["/usr/sbin/sendmail", "-t", "-oi"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
      emailOutput = emailProcess.communicate(msg.as_string())
  
      umountProcess = subprocess.Popen(umountCommand)
      mountData, mountError = umountProcess.communicate()
      print "Done..." #for GUI...
      
  else:
      msg = MIMEText(logs)
      msg["From"] = "python@parko.xyz"
      msg["To"] = "phaseform@gmail.com"
      msg["Subject"] = "rsync did not run"
      emailProcess = subprocess.Popen(["/usr/sbin/sendmail", "-t", "-oi"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
      emailOutput = emailProcess.communicate(msg.as_string())
      
  """
  -moove over script to lights
  """

  Currently I have a problem with rsync where it won’t stop when the drive is full..? this is compounded by my POS Telstra cable modem – which the RPi is connected to – which hangs on my SSH connection all the time… I’ll be swapping it out for an old Motorola Surfboard modem soon. Despite this rsync bug I’m testing with:

  sudo python "/home/ubuntu/gpio/DelugeRsyncEmailGUI.py"

  as from here and here.
  

  

  I added an rsync command before any writing is done to the USB, to remove any partial files, in case the USB is totally full, as here.
  Still need to add in flashing for the LED and other better indications of status.

• Using cp instead of rsync due to an rsync bug where it doesn’t stop syncing when the target is at capacity. Now my problem is that there are a bunch of files with size = 0 which occur when the USB device reaches capacity. Although I think that may be the only problem with this method.

  #!/usr/bin/env python
  import time
  import RPi.GPIO as GPIO
  import subprocess
  import os
  from email.mime.text import MIMEText
  import datetime
  GPIO.setmode(GPIO.BOARD)
  
  GPIO.setup(12, GPIO.OUT) #Red LED for fauilure
  GPIO.setup(13, GPIO.IN, pull_up_down=GPIO.PUD_DOWN) #rsync button
  GPIO.setup(16, GPIO.OUT) #rsync blue LED
  GPIO.setup(22, GPIO.IN, pull_up_down=GPIO.PUD_DOWN) #used to keep CPU usage low
  
  mountCommand = ["mount", "-L", "SneakerNet", "-o", "uid=ubuntu,gid=deluge,dmask=002,fmask=113", "/media/SneakerNet"]
  #mount -L SneakerNet -o uid=ubuntu,gid=deluge,dmask=002,fmask=113 /media/SneakerNet
  umountCommand = ["umount", "/media/SneakerNet"]
  mkdirCommand = ["mkdir", "/media/SneakerNet/Done"]
  rsyncCLEANCommand = ["rsync", "--delete", "-rptgo", "--existing", "--ignore-existing", "/media/Data/Deluge/Done/", "/media/SneakerNet/Done"]
  #rsync -rptgovh --delete --existing --ignore-existing /media/Data/Deluge/Done/ /media/SneakerNet/Done
  cpfilesCommand = ["cp", "-nvpR", "/media/Data/Deluge/Done", "/media/SneakerNet"]
  #cp -nvpR /media/Data/Deluge/Done/ /media/SneakerNet/Done
  
  def rsync_callback(channel): #rsync sequence
      GPIO.output(16, 0) #blue LED on.
      GPIO.output(12, 1) #Red off.
      if os.path.ismount("/media/SneakerNet/") == 0:
          MountProcess = subprocess.Popen(mountCommand, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
          MountData, MountError  = MountProcess.communicate()
          if len(MountData) == 37:
              mountfail = 1
              logs = "No USB with label SneakerNet found."
              GPIO.output(12, 0) #Red on
          elif len(MountData) == 0:
              logs = "Mounted"
              mountfail = 0
          else:
              logs = "Coding error or no su maybe?)"
              mountfail = 1
              GPIO.output(12, 0) #Red on
      else:    
          logs = "Already mounted"
          mountfail = -1
      
      if mountfail < 1:
          if os.path.isdir("/media/SneakerNet/Done") == 0:
              subprocess.Popen(mkdirCommand)
              logs += "\nMade Directory /media/SneakerNet/Done"
          rsynccleanProcess = subprocess.Popen(rsyncCLEANCommand, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) #clean Done Dir in case disk is totally full..
          rsyncData, rsyncError = rsynccleanProcess.communicate()
          today = datetime.datetime.now()
          with open('/media/Data/Deluge/RsyncTempLog.txt', 'w') as rsynclog:
              rsynclog.write("\nDate:")
              rsynclog.write(str(today))
              rsynclog.write("\nLogs:\n")
              rsynclog.write(str(logs))
              rsynclog.write("\nNow for cp:\n")    
          cpProcess = subprocess.Popen(cpfilesCommand, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) 
          cpData, cpError = cpProcess.communicate()
          #rsynccleanProcess = subprocess.Popen(rsyncCLEANCommand, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) #remove partials
          #rsyncData, rsyncError = rsynccleanProcess.communicate()
          with open('/media/Data/Deluge/RsyncTempLog.txt', 'a') as rsynclog:
              rsynclog.write(str(cpData))
              rsynclog.write("\ncp done ^.\nsend email and finish (hopefully...)\n")
          rsynclog = open('/media/Data/Deluge/RsyncTempLog.txt', 'rb')
          msg = MIMEText(rsynclog.read())
          rsynclog.close()
          msg["From"] = "python@parko.xyz"
          msg["To"] = "phaseform@gmail.com"
          msg["Subject"] = "RPi sync log"
          emailProcess = subprocess.Popen(["/usr/sbin/sendmail", "-t", "-oi"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
          emailOutput = emailProcess.communicate(msg.as_string())        
          umountProcess = subprocess.Popen(umountCommand)
          mountData, mountError = umountProcess.communicate()
          GPIO.output(16, 1) #blue LED off.
      else:
          msg = MIMEText(logs)
          msg["From"] = "python@parko.xyz"
          msg["To"] = "phaseform@gmail.com"
          msg["Subject"] = "rsync did not run"
          emailProcess = subprocess.Popen(["/usr/sbin/sendmail", "-t", "-oi"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
          emailOutput = emailProcess.communicate(msg.as_string())
          
      
  GPIO.add_event_detect(13, GPIO.RISING, callback=rsync_callback, bouncetime=300)
  
  try:
      GPIO.output(12, 1)
      GPIO.output(16, 1) #blue LED off.
      GPIO.wait_for_edge(22, GPIO.FALLING) #never going to happen
  
  except KeyboardInterrupt:
      GPIO.remove_event_detect(13)
      GPIO.remove_event_detect(22)
      GPIO.output(16, 1) #blue LED off.
      GPIO.cleanup() # clean up GPIO on CTRL+C exit
      
  GPIO.output(16, 1) #blue LED off.
  GPIO.remove_event_detect(22)
  GPIO.remove_event_detect(13)
  GPIO.cleanup() # clean up GPIO on normal exit