Pessimism and Optimism - Finding the balance

Pessimism is often seen as the negative in many situations, however, is too much optimism just as bad?

In a uncharacteristic departure from my usual blogs on technical problems, and possible solutions, I'd like to offer some food for thought on the social factors that effect projects, timelines and expectations. 

As someone who tries wherever possible to be optimistic about whether or not I can deliver outcomes to the customer, I sometimes wonder whether or not being too optimistic is detrimental. 
      For example if I say that I'll get something done for a specific date (the optimist in me) but it turns out that it's a larger body of work than I originally thought, I'd rather go above and beyond in order to meet the date than let people down. Now, I'm not saying that doing extra to meet deadlines is incorrect, far from it, but if it's happening constantly it may mean I am not learning from this experience. Am I masking an underlying problem that I didn't understand the work to be done? Or that I am poor at estimating the time it needs to complete a body of work? Is optimism in these cases taking precedence of realism?

Enter pessimism. Should I be more pessimistic about what can be done in order that peoples expectations are met without me having to constantly do extra to get it done? Or will this lead to bad relations with the people that you are working with?

Like everything it's the balance that counts. Without a healthy dose of pessimism I think that our promises to get work done will end up in letting people down or burn out in trying to make up for poor estimation. In contrast without optimism we would not be able to have drive to reach beyond our current situation and make things better than the status quo. 
      Therefore, I think the balance in the two end up being realism - the ability to deliver the things that you say you'll deliver in a time frame, that's achievable in your working day. I'd be really interested in what others think on this topic, whether this rambling is right or wrong.

Let's cover pragmatism another day...

AWS RDS MySQL SSL

Following my previous post of SSL with ELB's and instances I'd like to write a quick post about SSL and RDS MySQL. This is worth another blog post because it behaves slightly differently than your normal apache http or tomcat servers. 

This is because the SSL is enforced at the user level and not at the server port level. Typically when you disable insecure traffic to a particular server you will disable the port that it is listening on for insecure traffic - such as port 80 for the apache web server. 

MySQL is different, you still connect through the default port (3306), or whatever port you have configured it to run, but the difference is the way you connect. In order to ensure secure communication you must first create a user which requires SSL to connect:

First connect to the RDS instance as the root user:

mysql -h myinstance.123456789012.us-east-1.rds.amazonaws.com -P 3306 -u root -p

You can then create users in the specific way:

GRANT SELECT, INSERT, UPDATE, DELETE on db_name.* to 'encrypted_user'@'%' IDENTIFIED BY 'suprsecret' REQUIRE SSL;FLUSH PRIVILEGES;

You can now test this with the mysql client that you used earlier to connect to the database earlier:

mysql -h myinstance.123456789012.us-east-1.rds.amazonaws.com -u encrypted_user -p --ssl_ca=mysql-ssl-ca-cert.pem --ssl-verify-server-cert

The above should prompt you for the password that you set up above, and allow you to connect to the database securely. 

This is the database server end complete. The next part is to configure your application to use SSL to connect to the database securely. There are a number of ways in which you can complete this, for this example I'm going to configure a java application.

For this example we're going to configure the JDK to allow the secure connection. There are a number of options here, including application container specific configurations but this way has the advantage that all java applications (container or otherwise) will be able to connect.

First get the RDS MySQL server certificate from AWS:

wget https://rds.amazonaws.com/doc/mysql-ssl-ca-cert.pem

Now import this into the java trust store (replace $JAVA_HOME as necessary):

$JAVA_HOME/bin/keytool -importcert -alias rds -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeme -file ./auspost-root-ca.cer -noprompt

Connecting to the MySQL database is the same as before however you now specify three new options in your connection string which connect using SSL:

jdbc:mysql://myinstance.123456789012.us-east-1.rds.amazonaws.com:3306?useSSL=true&verifyServerCertificate=true&requireSSL=true

For brevity I will not include the rest of the code that you use in java in order to connect.

AWS and SSL - ELB to instance, Instance to ELB

At a glance

Here's the top tips to remember when attempting to enable SSL from an ELB to an instance and from that instance to another ELB:

• ELB's don't like backend self signed intermediate/root certificates, they need the actual certificate that the instance server is presenting
• Instances are quite happy to use the root certificate when connecting to a server presenting a signed certificate.
• Openssl and curl are your friends for testing (explained in detail below):

openssl s_client -connect <elb_dns>:443 -CAfile server.crt

• You can redirect your ELB listeners to listen on an un-secure port but communicate to the instance securely which is handy for testing
• Use ELB health checks with SSL in order to get continuous verification that SSL is working

In the detail - What we're attempting to achieve

SSL (TLS) is the standard for encrypting traffic between a client and a server. One of the best explanations that I've seen about it is here so I'll leave you to read through (I know I found it useful) and get an overview of SSL. 

In this exercise we'll be attempting to enable encrypted traffic from an ELB to an instance and from that instance to another ELB. For example say if I have a web tier and an application tier, I would typically have an ELB in front of each, therefore I need to configure both front and back end certificates on each ELB and also the instances.

I'll be using a certificate which has been signed by an internal root CA, as it's a little more complicated and more like a scenario that you'll be presented with when trying to complete SSL with your own server. Openssl have docs on creating signing requests and self signed certificates.

ELB configuration

A great guide for creating HTTPS ELB's is already defined by the kind people at AWS so I'll only concentrate on the parts that tripped me up.

Front End

Front end certificates in AWS are stored in IAM, allowing you to choose them when creating new ELB's. This is pretty smart as you'll typically use the same certificates for many ELB's. Here, it's just a matter of selecting an existing certificate or uploading a new one per the guide.

Back End

This was a real pain. I assumed that ELB's would work in the same way that the bundle file would in your OS, that is, that you could upload the root certificate to the backend of the ELB and it would be enough to work with the certificate being presented with by the instance. However this was not the case, so my number one tip, is to forego trying the certificate chain and just upload the certificate that is being presented by the instance, uploading the certificate to the backend is describe in the the AWS guide.

Testing

Finally we get to do some testing. I completed this using openssl to check the front end certificate and then curl to test getting a page from my web server once it was configured (see below).

openssl s_client -connect <elb_dns_name>:443 -CAfile /tmp/root.crt

This will connect to your ELB and verify that the root certificate you have specified locally will work with the certificate being presented by your ELB. You should get an OK if everything is configured correctly.

Instance Configuration

In this case I'm going to go with a simple example and use an apache web server configuration for SSL. There are numerous detailed posts out there about configuring apache for SSL so I'll go over the very minimum that will pertain to the AWS ELB specifics. 

Front End

Your instance (in this case apache) will need to be configured to present a SSL certificate to the client (the ELB). In order to do this edit the httpd.conf or vhost conf (more info here) that you have configured to listen on 443 to have these values:

SSLEngine on
SSLCertificateFile /path/to/www.example.com.crt
SSLCertificateKeyFile /path/to/www.example.com.key

As we learned in the explanation of SSL, the certificate, which is just a fancy public key, only works correctly with a corresponding private key file in order to decrypt the data that the client is sending to the server. After restarting the apache server you should be able to view the certificates that the server is presenting:

openssl s_client -showcerts -connect localhost:443

Back End

This is where the configuration is a bit easier and you can generally use a root certificate as opposed to the certificate that is being presented by the ELB. The advantage of this is that the root certificate will generally be valid for longer meaning you don't have to change them as often. 

For this example I'm importing the root certificate to the operating system bundle file that most tools use by default, including curl. Working with RHEL 6.5 we can complete the following command to import the certificate:

openssl x509 -text -in /tmp/server.crt >> /etc/pki/tls/certs/ca-bundle.crt

This can then be verified by the following command:

openssl verify -CAfile /etc/pki/tls/certs/ca-bundle.crt /tmp/server.crt

Testing

Front end testing of the apache web server can be completed by redirecting your un-secure (port 80) listener on your web ELB to talk to the secure port of your instance, like the following:

You can then connect to you ELB via http and know that it's connecting to your instance securely. This is just an option when you want to test individual configurations.

For backend, not only will you want to verify that the certificate is installed correctly (using openssl verify), but you'll also want to check that using the updated bundle to connect to the ELB will work correctly, this can be done with the following:

openssl s_client -connect <elb_dns_name>:443 

This should return you a bunch of text (including the certificate presented by the ELB) but at the bottom you should get an OK if everything is configured correctly. 

Finally: End to End testing

Considering that you'll have a page on your web server (like an index.html file or something) you can use curl to get the file securely through the ELB. I completed this on a VM that I had already installed the root certificate in the ca-bundle with the following command:

curl -v https://<elb_dns_name>/index.html

If everything is configured correctly then you should not notice anything different that using simple http, and you should get your page displayed on the command line.

Added Extras

Constant SSL certificate Health Check

One tip that I got from a colleague of mine is to configure your health check on the AWS ELB to use SSL in the health check, meaning that you'll get continuous verification that the SSL certificate is valid between your ELB and instance. This is particularly useful when considering that certificates expire:

Verify ssl key matches certificate

Another handy thing to verify, if you're not generating the public/private key yourself, is to make sure that the certificate matches the key. This is explained very well in another article.