Blog

Cloud Automation for Crate Tutorial

 

Crate ( https://crate.io ) is a distributed SQL solution, for real time distributed queries across a cluster. Crate is based on the Lucene / Elasticsearch ecosystem with built-in support for binary objects (BLOBs).

 

Setting up an cluster might be time consuming depending on the  number of instances in the cluster. This is why we provide an automation mechanism for setting up a fully functional Crate cluster using Amazon’s CloudFormation.

 

All you have to do is access a web page, select some options in the page and the CloudFormation template will be automatically generated for you.

 

Create an Amazon stack using this CloudFormation template and you’re done. In moments you will have your Crate cluster running.

 

The duration of the cluster creation and cluster instances discovery depends on the instance size you choose and the number of instances in the cluster. Choosing instance size with more computing power means that software will be installed faster. Also, if there are many instances in the cluster then the whole discovery process will take longer due to the fact that all nodes need to finish installing the software so that they are fully available.

 

What you need in order to use our mechanism:

 

 

Below are the steps you need to take for the full process:

 

  1. Access http://www.cloudwhiz.com/total-cloud-automation/crate/ . You should see the following on your screen:

 

crateblog1

 

 

  1. Select the desired Amazon EC2 instance size (default is t1.micro). Bear in mind that while t1.micro instance does qualify for free tier it will not allow ephemeral storage.

 

crateblog2

 

  1. Select the number of availability zones over which the instances should be launched:

 

crateblog3

 

  1. You can also choose to have more than one instance within the same availability zone:

 

crateblog4

 

  1. Type in a name for your EC2 instances that will be part of the cluster.

 

crateblog5

 

  1. If a different instance type than t1.micro has been chosen, then you can choose how the instance storage should be ephemeral or EBS:

 

crateblog6

 

and also, the storage size:

 

crateblog6.1

 

  1. In case the EBS storage type has been selected, there is the possibility to spread the crate database across two partitions:

 

crateblog7

 

  1. If there is the case, access to the Crate cluster machines can be restricted to a certain IP or IP class:

 

crateblog8

 

Be advised that if access is not restricted anybody will be able to connect to your Crate cluster!

 

If you are not sure what is your IP address, you can find that pretty easy by accessing https://www.whatismyip.com/ . The /32 that you see after the IP address in the example above means that we are restricting the access to that one single IP ( if you are interested in more details on what netmask means you could try this link: http://unixwiz.net/techtips/netmask-ref.html )

 

  1. Regarding the Crate version itself, you have the possibility to choose between the stable and development version to be installed:

 

crateblog9

 

  1. Once you’re happy with the settings just click on the “Generate” button and your Amazon CloudFormation template will be created and a download link will be provided to you:

 

crateblog10

 

  1. Click on the “Donwload” link in the page and save locally the CloudFormation template.
  2. Next, login into your Amazon Web Services account and go to the CloudFormation page, from the “Services” menu:

 

crateblog12

 

and you should be taken to a page that should look like this:

 

crateblog12.1

 

  1. (Optional) If you haven’t already done this, now is the time to add SSH keys into your AWS account ( detailed explanations on the process can be found here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html )
  2. Click on the “Create Stack” button within this page
  3. Within the page that is shown add a name for the stack that is to be created:

 

crateblog15

 

  1. In the “Template” section within the same page, click on “Choose file” button and navigate to the folder where you saved the CloudFormation and select the file

 

crateblog16

 

and once the dialog closes, click the “Next” button in the right side of the page.

  1. Next specify the name of the SSH key that should be used for the instances and, also, here you have the possibility to change the IP address range that is allowed to connect via SSH to the hosts. You may observe that the default value in the field is the one you have filled at step 8. Click “Next”

 

crateblog17

 

  1. In the next page you can add additional options for the instances that will be started:

 

crateblog18

 

  1. After clicking on the “Next” button you will be presented a review page of the stack to be created.

 

crateblog19

 

  1. Once the stack creation is started you can watch the events in the overview page:

 

crateblog20

 

  1. Once the autoscaling group has been created within the stack creation run, you can see in the EC2 section a new host/hosts and, by connecting to that host on port 4567 you will be able to see the installation progress:

 

crateblog21

 

  1. At the end, when all nodes in the Crate cluster are up and running, you will be presented a link to the Crate web interface.

 

 

 

 

 

 

 

 

.