How to run the Build Process in the Cloud with Teamcity and Amazon EC2

Roy Osherove did a great post on how to run Teamcity in the Cloud. You can find this post here. A while ago I did the same with LieberLieber, a Software Company from Vienna famous for their Product Amuse. The goal was to give additional flexibility to their build process. They use Teamcity for their Build Process and we the problem is that they sometimes need more computing power if they have a peek in projects. So we looked at the possibility of hosting additional Agents in the Cloud. Their Servers are still on-premise, but now additional Servers are in the Cloud for their Build process – and that’s on demand. They start a new Server once they have too much Build processes in the queue.
Roy and TeamCity as well provide a really great How-To on implementing this task. However, we discovered some minor problems. All TeamCity Installations for LieberLieber are Windows Servers as their projects are .NET Projects. We configured everything as described in the blogs, however, the Windows Firewall didn’t allow connections over Port 9090 – which is essential for TeamCity. If you configure TeamCity on Windows, make sure you add this Rule to the Windows Firewall:

After this was done, we could move on with the TeamCity build. For this first step we used the Windows Micro Instance at Amazon EC2. In the Screenshot below you can see that the Agent is up&running ;) .

Another Problem is the combination On-Premise <> Amazon EC2. The Build takes very long as the Agent needs to get the data from the Repository – which is On-Premise. In order to improove Performance, the Repository itself should be in the same Availability Zone as the Agent(s) and the TeamCity Installation. To compare the Performance of the Build Process, we told TeamCity not to Cleanup after a Build. This means that the Agent loads the Sources only when new Sources are available.

Once we could run the build at -almost- the same conditions, we were able to see the performance of the micro instance – and this was very impressive:

ip-0AE4F5A7 is the Amazon EC2 Server with the TeamCity Agent on it. As you can see, the Agent running in the Cloud was faster. Of course, this is just one run. To get a good comparision more runs need to be done. But imagine one thing: it is only 0,35$ per Day – the Agent needs to be available for 10 Hours per day. If more machines are required, it is absolutely possible to start more machines.

Data in the Cloud: SQL? NoSQL!

In recent years, one trend has become increasingly popular: NoSQL Databases. Databases built without relations, but made for high scalability. But is it really that “good” to go for NoSQL Databases? Doing so means removing the consistency that SQL Databases offers. Isn’t that a big tradeoff or is it worth going for more NoSQL Databases? What are the opportunities, advantages of NoSQL and why should you definitley keep an eye onto it? Join me for a tour on NoSQL. A world without relations, no joins and pure scalability!
A key disadvantage of SQL Databases is the fact that SQL Databases are at a high abstraction level. This is a disadvantage because to do a single Statement, SQL often requires the data to be processed multiple times. This, of course, takes time and performance. For instance, multiple queries on SQL Data occur when there is a ‘Join’ operation. Cloud Computing environments need high-performing, and highly scaleable databases.
Normally, SQL Databases have the advantage over NoSQL Databases to have better support for Business Intelligence. In most cases, Business Intelligence is not necessary in NoSQL Databases. If we look at the big Platforms in the Web like Facebook or Twitter, there are some Datasets that do not need any relations. The risk in that is the fact that there might be some kind of “inconsistency” across various  tables. The challenge for NoSQL Databases is to keep the data consistent. Imagine the fact that a user deletes his or her account. If this is hosted on a NoSQL Database, all the tables have to checked for any data the user has produced in the past. With NoSQL, this has to be done by code.
A major advantage of NoSQL Databases is the fact that Data replication can be done more easy then it would be with SQL Databases. As there are no relations, Tables don’t necessary have to be on the same servers. This allows a NoSQL Database a much more performant way to partition the data. This fact also applies to tables itself. If a table gets very big, this Table can be distributed over various servers easily. Again, this allows better scaling than SQL Databases. Don’t forget: scaling is one of the key aspects in Cloud Computing environments.
To fully understand why NoSQL Databases will be the future for Cloud Computing data, it is important to understand how Cloud Computing works. Cloud Computing Plattforms are made for a great number of people and potential customers. This means that there will be millions of queries over various tables, millions or even billions of read and write operations within seconds. SQL Databases are built to server another market: the business intelligence one, where less queries are executed.
Another disadvantage of SQL databases is the fact that there is always a schema involved. Over time, requirements will definitely change and the database somehow has to support this new requirements. This can lead to serious problems. Just imagine the fact that your application now needs two extra fields to store data. Solving this issue with SQL Databases might get very hard. NoSQL databases support a changing environment for data and are a better solution in this case as well.
However, it is up to the Use-Case to identify if you want a NoSQL approach or if you better stay with SQL

Error in AWS SDK for PHP: cURL error: SSL certificate problem

When using the AWS SDK for PHP, you might run into a SSL certificate problem. This problem could look like the following:
 
//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js //
Fatal error: Uncaught exception ‘cURL_Exception’ with message ‘cURL resource: Resource id #18; cURL error: SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (cURL error code 60). See http://curl.haxx.se/libcurl/c/libcurl-errors.html for an explanation of error codes.’ in C:Program Files (x86)PHPv5.3pearAWSSDKforPHPlibrequestcorerequestcore.class.php:843 Stack trace: #0 C:Program Files (x86)PHPv5.3pearAWSSDKforPHPsdk.class.php(1034): RequestCore->send_request() #1 C:Program Files (x86)PHPv5.3pearAWSSDKforPHPservicesses.class.php(399): CFRuntime->authenticate(‘SendEmail’, Array) #2 C:inetpubwwwrootwordpresswordpresswp-contentthemestwentyelevensingle.php(23): AmazonSES->send_email(‘mario.mh@cloudv…’, Array, Array) #3 C:inetpubwwwrootwordpresswordpresswp-includestemplate-loader.php(43): include(‘C:inetpubwwwr…’) #4 C:inetpubwwwrootwordpresswordpresswp-blog-header.php(16): require_once( in C:Program Files (x86)PHPv5.3pearAWSSDKforPHPlibrequestcorerequestcore.class.php on line 843
The reason for this is that you might have misconfigured your Certificate Settings. To find that section, go to your “config.inc.php” File in your SDK’s Directory. The file is used to store your credentials and connection settings. Find the variable that says “certificate_authority”. If it is set to “false”, you have to specifiy the File that holds your Connection Settings. If you use the SDKs File (if you haven’t changed any SSL settings), change this value to “true”. Now you should be fine. Make sure that your SSL settings are secure and avoid “man in the middle” attacks!

Source Code for AWS and .NET Sample

The Source Code for the e-book is now available at Codeplex.
The Sample includes the following Services:

  • Amazon EC2
  • Amazon SQS
  • Amazon S3
  • Amazon DynamoDB
  • Amazon Elastic Beanstalk

The Sample is written in C# and uses Asp.NET MVC, Asp.NET Web API and AWS SDK for .NET

Cloud Computing: Praxisratgeber und Einstiegsstrategien

The Book “Cloud Computing: Praxisratgeber und Einstiegsstrategien” is written in German and gives an introduction to Cloud Computing. Core Topics are:

  • Elements of Cloud Computing
  • Technical fundaments of Cloud Computing
  • Architectural fundaments of Cloud Computing
  • Business
  • Law
  • Comparing common Cloud Computing Vendors such as Amazon, Microsoft, Google, IBM, …

The Book is available on Amazon here.

Big Data: Elements for Data Quality

Whenever we talk about Big Data, one core topic is often not included: Data Quality. If we Data, all the Data doesn’t really help us if the data quality is poor. There are several key topics that data should contain in terms of quality.
Relevance – Data should contain a relevant subset of the reality to support the tasks within a company.
Correctness – Data should be very close to reality and correct.
Completeness – There should be no gap for data sets and data should be complete as possible.
Timeliness – Data should be up-to-date.
Accuracy – Data should be accurant to serve the needs of the enterprise.
Consistency – Data should be consistent.
Understandability – Data should be easy to interpret. If it is not possible, data should be explained by metadata.
Availability – Data should be available at any time.
 

Cloud Computing for CIOs: Elements of IT Security for Cloud Computing

We often hear executives in various companies talk about IT security. Especially when it comes to Cloud Computing, security is a major topic. Many IT leaders and CIOs are afraid of Cloud Computing when it comes to security. To find out about security in the Cloud, we have to discuss what elements of IT security are necessary to Cloud Computing. The 6 elements of IT security are called “ACACIA”.

ACACIA as the main factors for IT Security

ACACIA as the main factors for IT Security. Photo by Donald Hobern


Availability
Availability says that all Elements within an IT landscape must be available. Users shouldn’t be limited by outages or similar things. If we talk about the Cloud, we need to handle this with failover systems, such as different availability zones.
Integrity
This means that the IT landscape is fully operational and there are no alien servers – which, for instance, could have been added by competitors or hackers. It could also happen that some malware is installed. Therefore, it is necessary to protect your Cloud-based infrastructure, as you would do it with your on-premise devices.
Confidentiality
Only those people, that fulfil certain requirements, are allowed to administer or use certain services. If a user is not part of the HR department, the user should not use this section in the intranet.
Anonymity
In many cases, users should have the possibility to use certain anonymity sites. For instance, when it comes to voting, nobody wants to be listed in a Database. Therefore, anonymity must be provided for certain tasks.
Authenticity
If two systems communicate with each other, it is necessary to know that sender as well as receiver are in the correct position. An error to authenticity could occur, if an enemy system knows the means of communication and “catches” messages, modifies them and handles them over to the intended receiver.
Commitment
This means that the IT-System is lawful and everything is done in a way that local as well as international law requires.
 

Cloud Computing for CIOs: Understanding the Innovation Level of a Technology

To manage Technology in a good manner, CIOs need to understand what type of Technology they deal with. This is also very important when it comes to Cloud Computing. We have different technology topics and levels with Cloud Computing such as Software as a Service, Platform as a Service and Infrastructure as a Service. But how mature and how advanced is a technology? To figure that out, we have 4 different technology types.
Basic Technology
The first technology type is called “basic technology”. This means that it is a technology that has been around for a while, is very mature. However, there is only very little to almost no innovation level involved. Technologies associated with Cloud Computing usually don’t fall in this type of technology since there is still a lot of innovation going on with Cloud Computing. However, we could call Virtualization a basic technology.
Key Technology
Next, there is the “key technology”. This technology type also refers to an existing technology, but with a lot of innovation in going on in this area. We could call “Software as a Serivce (SaaS)” a key technology. It is possible to use key technologies in Enterprises.
Pacemaker Technology
We call a technology type a “Pacemaker Technology” if the technology is still evolving and we expect to have a high level of innovation affiliated with it. A pacemaker technology might change quickly and this means that you have to prepare for that. Infrastructure as a Service is currently somewhere between a Key Technology and a Pacemaker Technology. The same applies to Platform as a Service. They are to “young” to be called a Key Technology but also somewhat too mature to be called a Pacemaker Technology.
Future Technology
A future technology is a technology that is not yet here. We can call a technology a future technology if it is about to arrive within the next few years. There is nothing like that in Cloud Computing. Cloud Computing is not a future technology, it is a present technology. We could define LTE a future technology. It will come in the next years, but it is not available for the “critical mass” yet. A future technology will still change a lot. If we look at LTE right now, we can clearly see that there are no real standards available.

Cloud Computing for CIOs: the Role of your IT within the company

If we look at how your company is built from the inside, it is possible to get some information on your company’s IT power. Research on MIS (Management of Information Systems) calls this topic “the strategic role of information management”. This topic knows 4 major strategy types, which are listed below.

Company IT potential by function

Company IT potential by function


Let us discuss each company type and how it fits into cloud computing.
Type 1: Support. In this company type, the current IT potential as well as the future IT potential is low. This means that IT is not very important and will never be important. IT basically consists of some minor operating tasks, whereas strategic and administrative tasks are not included. IT costs are basically low and will stay low. Cloud Computing can help these company types by reducing the number of software licences such as for Text editing by outsourcing that to the Cloud. It could be handled by Google Docs or a similar SaaS Application.
Type 2: Fabric. Fabric means that the current state of the IT potential is high, but it is reduced significantly. This often happens when a company is sold to another company and the new owner will handle IT. Major investments are postponed and the only important thing is to keep IT up and running until it is handled by the new department. New projects (if there are any) could be done by Cloud Platforms such as IaaS. PaaS might not be that good since projects are likely to be replaced in the near future. In my opinion, PaaS Projects are for scalability (IaaS also). However, IaaS are great to do some timely limited Compute topics.
Type 3: Breakthrough. In this IT type, the potential is currently low but it will get important in the future. IT spendings will increase significantly and the company will focus on IT as strategic opportunity. To make most out of the budget, spendings on Cloud Platforms of any type are to consider. New Projects can be done with PaaS Platforms as they offer additional benefits in terms of programming model and scalability.
Type 4: Weapon. The current IT potential is high and it will stay high in the future. Depending on what the company offers and the level of security associated with it, a major switch to cloud computing is possible. However, it must be evaluated what services need to stay within the company (private cloud) or what services can be outsourced to a public cloud.
 

Cloud Computing Weekend Reading: AWS, Forbes, IBM, Open Source

Still can’t get enough information about Cloud Computing? Here is the weekend reading list for interesting Topics about Cloud Computing. Have a nice (and cloudy) weekend 🙂
//
http://cloudvane.azurewebsites.net/wp-content/uploads/2012/10/show_ads.js

  • Google search to include your private Gmail soon? – In this Blog by Computerworld the possibility of Google to include Gmail Data into search is discussed. There is some interesting Feedback by some authors. Read more here.
  • Only 16% of people know what the Cloud is. Do you agree? Is Cloud Computing not well known enough? Most believed it was about drugs, pillows, the weather, or toilet paper. Read more on Forbes about it.
  • 5 considerations when moving to the cloud. IBM explains what we have to consider when moving to the cloud. The 5 Tips are Monitoring, Security, Performance, Vendor-Lockin and Migration. Read more about it here.
  • Amazon Web Services boss Andy Jassy on competition, price wars, and getting big. Read the interview with AWS boss Andy Jassy on his thoughts about Cloud Computing and what is going on right now in this sector. The interview can be found here.
  • The Role of Open Source in Cloud Computing Innovation. Interesting article about the role of Open Source for Cloud Computing. The article can be read here.