Mission: hourly poll of a website
...to find out if the Kitteh is available for adoption and immediate email notification if Kitteh is found.
Estimated time to complete: between 15 minutes and hours (depending on setting up your EC2 instance, SES service, etc.)
Skills: Amazon EC2 setup, SSH, centos yum, bash, wget, cronjob
Amazon Free services tier
If you have an Amazon EC2 instance running (e.g. EC2 Linux Micro Instance in Free Tier = centos!)
(And you're not running over the GET/POST upload/download free tier bandwidths)
(If you don't know how to setup a quick Amazon Linux Micro Instance in the free tier search this blog for more info)
- Sign up for SES (then receive a verification email for your Amazon AWS Account)
- Account Security Credentials (for AWS access identifiers)
Use nano or vi to create a file "aws-credentials" (Amazon's sample below)
Download the example perl scripts via: http://aws.amazon.com/code/Amazon-SES
wget http://aws-catalog-download-files.s3.amazonaws.com/AmazonSES-2011-02-02.zip unzip AmazonSES-2011-02-02.zip chmod +x /home/ec2-user/*.pl /home/ec2-user/bin/ses-verify-email-address.pl -k aws-credentials -v [email protected]
Amazon EC2 Missing some perl
"Can't locate XML/LibXML.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/local/share/perl5 /usr/lib64/perl5 /usr/share/perl5 /usr/share/perl5 /usr/lib64/perl5 /usr/share/perl5 /usr/local/lib64/perl5/site_perl/5.10.0/x86_64-linux-thread-multi /usr/local/lib/perl5/site_perl/5.10.0 /usr/lib64/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl /usr/lib/perl5/site_perl .) at ./ses-verify-email-address.pl line 26. BEGIN failed--compilation aborted at ./ses-verify-email-address.pl line 26."
THANKS AMAZON! Using their preconfigured Instance means they don't have all of the Perl packages installed...
sudo yum install perl-XML-LibXML perl-digest-SHA
sudo yum provides */SHA.pm
tells me what other packages I might have missed...
sudo yum search perl-Digest
sudo yum install perl-Digest-SHA
What a difference a D versus d makes!
- sudo yum perl-libxml-perl libxml2-devel perl-IO-Socket-SSL libxslt-devel ?
sudo apt-get install libio-socket-ssl-perl libxml-libxml-perl
tail /var/log/maillogfor troubleshooting sendmail...
Verify an SES linked email address by running the perl script
/home/ec2-user/bin/ses-verify-email-address.pl -k aws-credentials -v [email protected]
Use the email account you gave above for verification and click on the link...
You have successfully verified an email address with Amazon Simple Email Service.
~/amazonses/bin/ses-send-email.pl -k ~/amazonses/bin/aws-credentials -s "Test AWS" -f firstname.lastname@example.org email@example.com,firstname.lastname@example.org < ~/kittysearch/result.txt
#/bin/bash wget -O ~/kittysearch/page1.htm 'http://adopt.hssv.org/search/searchResults.asp?task=search&searchid=&advanced=&s=adoption&animalType=2%2C15&statusID=3&state=®ionID=&submitbtn=Find+Animals' wget -O ~/kittysearch/page2.htm 'http://adopt.hssv.org/search/searchResults.asp?tpage=2&task=search&searchid=&advanced=&s=&animalType=2,15&statusID=3&state=®ionID=&submitbtn=Find+Animals' wget -O ~/kittysearch/page3.htm 'http://adopt.hssv.org/search/searchResults.asp?tpage=3&task=search&searchid=&advanced=&s=&animalType=2,15&statusID=3&state=®ionID=&submitbtn=Find+Animals' grep -i "bandit" ~/kittysearch/page1.htm ~/kittysearch/page2.htm ~/kittysearch/page3.htm > ~/kittysearch/result.txt if [ -s ~/kittysearch/result.txt ]; then # must move to the directory to use the SES.pm cd ~/amazonses/bin ./ses-send-email.pl -k ~/amazonses/bin/aws-credentials -s "Test AWS" -f [email protected] [email protected],[email protected] < ~/kittysearch/result.txt fi # http://docs.amazonwebservices.com/ses/latest/DeveloperGuide/ for full details about email.pl
Trigger the search with cron
i key to enter input in vi
55 * * * * /home/ec2-user/kittysearch/kittysearch.sh
escape key gets : then x to save and quit
Troubleshooting "Cannot locate SES.pm"
Running the script from a different directory or CRON gets the error: "Can't locate SES.pm in @INC"
cp /home/ec2-user/amazonses/bin/SES.pm /home/ec2-user/kittysearch
FIXED! must move to the directory in the script using cd first to have access to SES.pm
SES Message Limit
Yyou can send 2,000 messages for free each day when you call Amazon SES from an Amazon EC2 instance directly or through AWS Elastic Beanstalk. (Note bandwidth charges may still apply)
Apparently since 2011 there has come along infrastructure like page2rss and ifttt that makes these kind of custom solutions less helpful (unless you need customization!)