Because SSH and “for” loop doesn’t cut it

When we configure the amount of PHP processes that we spawn in our frontends we base this decision on the amount of CPU power of the server. This has a direct relation with the amount of memory, given that more CPU means more concurrent requests and also more memory usage. For automating the amount of PHP children that should be spawned we use a custom facter. A custom facter is a piece of Ruby code that is executed to return a fact for a node. The default facts are hostname, ip address, netmask, amount of memory, processors, kernel version, etc, and they’re very useful, but sometimes you need to extend it. In the example displayed below we create a custom fact called “tuenti_phpchildren” that will take decisions about the amount of PHP children spawned per server. I use an existing custom facter called “tuenti_memorysize” that returns the amount of memory installed in a server as an integer.

Facter.add("tuenti_phpchildren") do
setcode do
memtotal = Facter.value('tuenti_memorysize')
if memtotal > 20000000
phpchildren = "300"
elsif memtotal > 9000000
phpchildren = "200"
elsif memtotal > 5000000
phpchildren = "100"
else
phpchildren = "50"
end
phpchildren
end
end

Then we use the custom facter inside a template, to ensure the file contains the proper configuration for the daemon.

max_children = <%= tuenti_phpchildren %>

Automating your whole platform using Puppet is not something you’ll accomplish in three days (if you have a large fleet of servers with different roles as we have), but it’s a really worth the effort. Some advantages we got from moving our configuration management to Puppet are:

• No configuration drifts: Puppet is checking in the background that everything is fine and will revert to the desired state if something has changed (accidentally or on purpose).
• Become more agile: We are a fast-moving company that requires changes every day and we can’t continuously spend our time on repetitive tasks. Now we deploy changes to production within minutes having more control than we use to have in the past.
• Save time: We automate once and we could be able to easily bootstrap hundreds of nodes in minutes.
• Comply with industry standards: Some law enforcements like Spanish LOPD or PCI Compliance require a list of policies that must be met in all the servers from your company, like access controls, encryption ciphers used in SSL transactions, password management and more.
• Document itself: Puppet won’t excuse you from documenting your platform, but once someone reads a manifest they’ll already understand what that piece of code is doing. No more tedious explanations about what to do or what to execute for configuring a service.
• Share knowledge: Now we’re able to share our Puppet modules with other teams like QA or Dev Tools in order to align our development, testing, and production environments. Also, using different Puppet environments we can have different versions of a module in different stages like development, staging and production, while having a reasonable workflow: First you develop it, then you try it across some servers, and finally you deploy it to the whole platform.

Our current Puppet architecture is based in three Puppet Masters that bring configuration management to more than 1k nodes. Scaling this platform is as easy for us as adding extra nodes and adding them to the load balancing pool. Puppet also offers a reporting feature that can be used with the Puppet Dashboard in order to have a view at a glance of your platform. Given that all the reporting data is stored in a database you can integrate it with your internal tools or dashboard in order to detect nodes that are not applying configuration properly or are suffering unexpected changes.

To make the development of manifests easier, a project named Geppetto was born some time ago. Besides the tool itself, it offers a module for Eclipse that will make your life much easier. For hardcore people, you also have Vim syntax highlighting provided by PuppetLabs.

In summary, even if you only have one server you need to be able to reproduce its configuration. So if you have the opportunity, try some configuration management like Puppet and move out from sysadmin hell to operational bliss.