Getting started with logstash part 1

Statsd & Graphite

In my last post I mentioned statsd and graphite, these are 2 tools that I use in conjunction with logstash and they make a good team!

So the general idea behind running these utilities together is that you can get fancy graphs based off certain events, so for example it might be nice to see a longer term trend of IIS response times for a given website or something to that effect.

I won't go into how to set them up as there are a couple of guides around on how to do it, I can try and point people in the right direction if you have any questions however.

One important-ish thing to note is that if you wish to save a group/single graph (you most certainly will!) you will need to set up a django account when you configure the graphite web app. You can make this work after you have installed it if you decided to skip that step.

Here is how I have configured one of my statsd metrics:

statsd {
    type => "iis"
    tags => ["example.co.uk"]
    exclude_tags => ["_grokparsefailure"]

    increment => "live.example.iis.response.%{response}"
    timing    => ["live.example.iis.responseTime.", "%{timetaken}"]
  }

Events of type "iis" which have the tag "example.co.uk" and did not fail grok parsing are accepted by this output filter. Each time this filter recives an event the metric for that particular response code is incremented (So you could plot a graph of how many files you were missing, 404s, for example). the %{field} notation is used to indicate that this bit of text should be taken from the event itself.

You can have many of these output filters defined for events with different tags, so you can have metrics for different sites etc.

One thing that tripped me up was the storage aggregation configuration, to the lay-man this means that when your storage schema for carbon specifies that metrics should have thier precsion reduced a certain set of rules determine what gets stored as the new value for this time period. However, as the metrics I was tracking did not always have a value (null) I found that for the longer graph periods there was no data. A great post over at http://obfuscurity.com/2012/04/Unhelpful-Graphite-Tip-9 pointed me in the right direction on how to resolve this issue.

Logstash is a tool for doing fancy stuff with log files. Its main purpose is to help administrators/IT staff with the monitoring of logs, specifically it can be used to push logs to a central location where the logs are stored and indexed for later searching. A basic logstash setup (like the one I am implementing at work) consists of log shippers (web servers and database servers) and then one server where the logs are stored, indexed and searched (in my case this is a Ubuntu server).

More...

I have now completed the game and acted on the feedback that was given to me when we did the play testing session.

I think that I have produced a well rounded game with a unique concept considering that most of the submissions for the module were sidescrolling flash games built in stencil works. I am glad I chose to implement the game in HTML5 & JavaScript as it is better aligned to my Web Applications Development degree, I have learnt some good practices for general JavaScript development and more details on interacting with the canvas element, in addition to learning how game mechanics should work.

I am happy with my Highscore implementation as it uses an XML datasource and the ASP.NET MVC Web API, my only complaint is that the initial request to start the web application (After the app pool has been unloaded to conserve resources, because of a long wait inbetween requests) can be rather slow, however this will be less of a problem the more popular the game becomes. If I was self hosted (As opposed to shared) I could also extend the life of my app pool to ensure that it was not retired when waiting for requests. The game also has a complete ending which helps make the game feel finished.

More...

Yesterday we had the feedback/play test session. I recieved some useful feedback from students and lecturers alike, the main points people picked up on were:

• You cant tell what a station goods a station was buying or selling.
• It wasnt obvious when you ran out of money/cargo space when purchasing goods.
• You couldnt check what you had purchased in your hold - and for what price (I dont think I will be implementing price recording, as its part of the challenge to become a good trader!)
• It was not immediatley obvious (untill your first death) that the pirate base is in the top right and that its bad to go there untill you have the right level of gear.
• The highscore does not take into account the level of your gear/progress or how much "stock" you have in your hold.
• It can sometimes be difficult to get started with the trading if the economy is randomised unfairly (E.g Energy is in low demand and or expensive at the power plant).
• Lack of engaging story.
• No way of knowing the max quantity of the product.

To address these issues I have: More...

Yesterday I posted my first answer to a Stack Overflow question (granted it's not very exciting, but it's still an answer right?!), I have used the site on many occasions before, generally to look up the solution to a problem I have been experiencing, but have not yet posted a question. Stack Overflow is an excellent resource, something I hope to be able to contribute to, I can see how addicting gaining rep and answering questions will be! As exciting as gaining reputation will be I enjoy problem solving, so if I can help other people out as well then that's great!

My "flair":

Rather than just calling my game "Space Simulation Game", a catchy title as that is, I thought it would probably be a good idea to come up with a proper name, especially as I have completed the majority of the functionality. As the game is about trading and "ace" implies you should be good at it, I thought "Ace Traders" would be an apt name for my game. You can find the beta version of the game at spikewade.biz/university/spacegame. I have explained a few of the implementation details below.

More...

I have been setting up a little web api with MVC4 and entity framework for a university project, everything was running fine on my local development machine, however when I created the database on my host (123-reg), via scripting the database (From SQL Management Studio) and removing commands 123-reg doesnt accept, my application would try and create the database, which my database account doesnt have permission to do. This was very strange behaviour as the database was in a compatible state, DbContext.Database.CompatableWithModel() returned true, but EF still thought that the database didnt exist, DbContext.Database.Exists() returned false. To fix this I found the following post on Stack Overflow http://stackoverflow.com/questions/4558377/entity-framework-4-ctp-5-code-first-development which said that the Database Intialiser needed to be set to null. After applying this change and updating my remote site everything worked as expected.

I am still unsure why EF thought that the database didnt exist, but I am pleased it works now!

I would like my game to be available across the internet independant of platform, HTML5 and JavaScript dont rely on any plugins or codecs that a user might not have installed and so make a good choice for implementing a game. One possible downside to this pair of technologies is the immaturity of them compared with games developed in flash that have large user bases and lots of tutorials and other content, additionally flash have access to many different portals through wich the game can be marketed and monitised.

More...

As I mentioned in the first blog post my game is going to be set in space, therefore it makes sense if the design and aesthetics of the game are somewhat futuristic. I found a set of graphics released by Lost Garden http://www.lostgarden.com/2005/03/download-complete-set-of-sweet-8-bit.html which seem to meet my needs additionally I quite like the retro look incontrast to the very modern and impressive graphics used in S.P.A.Z. The graphics pack features several different objects that could be used as space stations, where goods can be bought and sold, in addition to some ships that I plan to use for the player's own ship. I hope to add some sound effects to the game after I have completed the majority of the programming, ambient music and sounds for actions like docking the ship and buying/selling goods should add to the atmosphere of the game.

More...

For my as yet unnamed game I plan to create a space simulator game that includes trading mechanics. The player will have to improve the capabilities of his ship through purchasing upgrades with Credits. Credits are earned by buying and selling goods between stations.

More...