Hitchkhiker’s guide to Topcoder, a Cloudspoker’s view

4 Replies

Topcoder was acquired by Appirio (Cloudspokes parent company) recently, which has ended on the recent merging of the two platforms. I think this makes sense because Topcoder is a much more mature platform, and the Topcoder team along with the community is already doing a great job merging the best of both platforms on the new awesome page. This said, I just completed my first Topcoder challenge, so I thought on writing my experience as it might be valuable for fellow Cloudspokers. So bear with me, don’t panic and don’t forget your towel!

I was a Cloudspoker for one and a half year, and I loved the platform. It had some flaws but it was very dynamic and fun to compete in. I am sure most of my Cloudspokes partners felt that way. When I registered for Topcoder, my first impression was something like this:

Sorry for the joke, but surely you get my point. UML editor, Assembly, Arena, Studio, SRMs… So many things! I am also sure most of my colleagues are feeling or felt this way at some point. However, once you start moving handles and pushing buttons, it starts to make sense. So let me expose my findings in the next five points:

Register fast!

Topcoder has a registration phase, so you have a couple of days to register on a challenge. After that, even if you are capable of code the solution on the remaining time, you are not allowed to do so. This may sound inconvenient at first, but I think I know the reason and it is for good.

It was really frustrating to check Cloudspokes challenges, see ten people registered and say “Ahhm, so crowded, better try the next one…” just to finally realized nobody really submitted and you should have registered. This solution addresses the problem along with the next point.

Think before registering

On Topcoder, you have a reliability score. And reliability bonus along with this score, which is higher when you submit every time you register. I think this is a cool way to avoid massive registering. And even if a challenge is crowded, you can know who is more likely to submit, just to take a look at the score.

One think I think it could be better is being able to view the participants of each challenge from the challenge page itself like on Cloudspokes. Now it is a bit hidden, you will have to list all challenges, click on the participants number link which will take you to the detail page, but I am sure this will be covered on the new design.

Make forums your friends

On Cloudspokes we had a single thread on the same page challenge to ask for clarifications, specification updates, etc. On Topcoder we will have to dig into the forum challenge. I personally find Cloudspokes approach is easier and more usable here, as you could just read everything from top to bottom, but in exchange we will get a better organized forums and subforums, with more configurable notifications.

Understand different phases

On Cloudspokes we basically had 3 official phases, being:

Register
Submission
Results

Adding an ad-hoc fourth one sometimes between submit and results, asking for fixes or clarifications. On Topcoder this gets a bit more complicated, I will try to explain each one.

Registration: As I said before, don’t wait till the submission deadline, or you will be too late cowboy!
Submission: This is basically the same. Big warning here, the platform currently only accepts a file, and each one overwrites the last one, so you will have to zip the solution. Also it does not accept URLs, so include them on the readme.
Screening: This is a filter prior to judge, be sure to read the screening scorecard to pass! There is the possibility to not passing screening for a valid solution just for configuration issues or tiny error. I think there should be a reasonable short period of time when you are able to fix your submission if it does not require major changes. However, when used carefully I think this might be useful for judges, I remember losing time judging really weird things back on Cloudspokes.
Review: This is the same as in Cloudspokes, you are judged by other community members, only difference is that this score itself will not earn yourself the victory. Warning: Unlike in Cloudspokes, judging is anonymous, so instead of your name and email, you will have to put a placeholder on the documents just before the final phase.
Appeals: Remember comments on the scorecards? Now you have the possibility to reply to the judge! Big Tip: I got a 87% initial score, and I was the only submitter through screening, so as a beginner I thought: “Nahhh, I don’t want to appeal, I’m fine”. Big mistake, because to pass the final review you will have to address all these issues brought by the judges, and this is your possibility to flag “false positives”, issues that really are not, in order to reduce the final list of issues.
Appeals response: Here the judges will validate your appeals, and then goes to a single judge to sum up every issue.
Aggregation: You will receive a final scorecard with all the found issues for you to fix.
Final fix: Here you will finish the submission so everything is perfect. That means final review will have to pass with 100%. This is very important and a big difference with Cloudspokes.
Final review: The judge doing the aggregation will finally review the application.
Approval: Here the copilot will validate the solution, and finally you will get your beloved prize!
Guarantee: This is a period of time when you might have to fix hidden issues to get the rest of the prize.

Pay attention to the details

I found the Topcoder review system a bit more rigid than on Cloudspokes. For example, I got complaints on the deployment guide format. There is a template, and you have to use it. Also all the new code has to be documented.

I hope these tips help you. Let me know if you have any other tips!

The ninja guide to hackatons

Sit, relax, think

愚公、山を移す – Nothing is impossible to those who have sufficient faith

My first reaction to the APIs I got was like “well, I’ve got no possibilities, so let’s enjoy DC and better luck next year!”. However, a few hours after spinning the wheel, I came up with some feasible ideas (Well, I must say we, credit to my girlfriend who really helped me here) that didn’t sound not so bad. So the tip here is not to be overwhelmed with the whole challenge, just relax, and start thinking. There is always a way to go through! After the brainstorming, I started proof of concept which guides us to the next point.

Start small, grow big

虎穴に入らずんば虎子を得ず – If you do not enter the tiger’s cave, you will not catch its cub

Even the most honored Sensei start small. I had never worked before with any of the APIs, so it was not clear at first what could be done with them. So I got a base project on a few minutes (we will talk about this later) and did three simple, really rudimentary web forms that were supposed to trigger the APIs in the most basic way and write some output to console. Just the fact that something was working, cheered me up and I got a nice point of view of what to do next.

Start with a supporting point

鳶が鷹を産む – A kite breeding a hawk.

On hackatons, time is money (literally) so you don’t want to go throwing shurikens all against your ninja path, just to your target. Don’t start from scratch and try always to have a really basic start point, even if it is just setting up a simple webserver with a page saying “Hello world”. I am a java guy, but I think new technologies like NodeJs can speed up development for prototyping, so I went with the MEAN stack project. You can have a working, database-connected app deployed to heroku in five minutes. Once you see the “this is the main page” message on heroku, morale goes up!

Do not reinvent the wheel, build the coolest car

亀の甲より年の功 – Years know more than books.

You should have tools in your ninja toolbelt which you could leverage on to do all the boilerplate code, so you can focus on the cool stuff you want to build. Nowadays, in the Github era, it is difficult not to find at least someone that have tried what you are trying to do, so look for code to reuse, and leverage on well know components. You need authentication? PassportJS got you covered. You are not an artist with css? Relay on Twitter Bootstrap. Just focus on the cool stuff your application is supposed to do.

There is always hope

七転び八起き – Fall down seven times, stand up eight

There would be some obstacles on your ninja path, that will drag you from achieving the final result. Just don’t let them drive you mad. Have you thought of all the possibilities? Is there an easier way to achieve your objective? At some point on the hackaton, I found myself installing custom libraries on heroku at 3AM to generate QR codes at the lowest level, which made me loose precious time with no result. But after relaxing for a bit I thought, “Hey, I’m in the cloud, why can’t I relay on it?” So I searched for QR generation APIs and was able to get the functionality I wanted in about 15 minutes. Sometimes is better to take one step back and think to get one further.

There is always hope

Divide and conquer

小打も積もれば大木を倒す – With many little strokes a large tree is felled

You will not see ninjas in action facing all enemies at the same time like in a Chuck Norris movie, instead, they take one target at a time with precision, then go to the next. You should write down in a paper all the targets you have, and start crossing them out once they are done. That way you will have a clear view on where are you in your path. Put the most important ones on the top of the list, the nice-to-haves on the bottom. I left some of the features I wanted on the application uncrossed, but I got a finished product, which is more valuable than a bunch of half implemented features.

So this are my thoughts, maybe they don’t fit everyone out there, but were for sure helpful to me. Let me know if this was useful to you!

AngularJs log in with LinkedIn

3 Replies

I was looking for a way to implement the LinkedIn sign in functionality with AngularJS, so I stumbled upon this article on LinkedIn documentation. The goal was to implement the button, but only using client side javascript. For that, we will use something called OAuth 2.0 implicit flow (a.k.a the client flow), here you can find more details about it.

Basically, we will register at LinkedIn as a developer and then get an API key. We will have to add a new application, and tell LinkedIn our domain (for example, for testing, http://localhost). After that we copy the value on API key field to use it on our application

I have a sample code to test the API on this github repo. The only set up is to go to index.html and put the API key like this:

<script type="text/javascript" src="http://platform.linkedin.com/in.js">
  //here goes the api key, and the callback function
  api_key: <here goes the api key>
  onLoad: onLinkedInLoad
  authorize: true
</script>

Then we start the application, go to localhost and click on sign in with LinkedIn:

After authorizing the app on LinkedIn, the OnLinkedInLogin will be triggered:

//execute on login event
function onLinkedInLogin() {
   // pass user info to angular
   angular.element(document.getElementById("appBody")).scope().$apply(
   function($scope) {
      $scope.getLinkedInData();
   });
}

Note the $apply method which is used to to execute an expression in angular from outside of the angular framework. That will trigger a function defined in our scope:

$scope.getLinkedInData = function() {
if(!$scope.hasOwnProperty(“userprofile”)){
IN.API.Profile(“me”).fields(
[ “id”, “firstName”, “lastName”, “pictureUrl”,
“publicProfileUrl” ]).result(function(result) {
// set the model
$rootScope.$apply(function() {
var userprofile = result.values[0];
$rootScope.userprofile = userprofile;
$rootScope.loggedUser = true;
//go to main
$location.path(“/main”);
});
}).error(function(err) {
$scope.error = err;
});
}
};
And we will be able to get the user info from LinkedIn.

This code could be much more the angular way, defining the functionality on a directive and making it much more generic, but is a valid start point for a integration.

Predicting Migration Times With Google Prediction API

2 Replies

The Challenge:
After a series of challenges on the GAMME Estimation tool, I decided to do a blog post proudly showing my development, but when the CloudSpokes team told me I could do a guest post, I took it even more seriously! When I came across the GAMME Estimation contest, I had recently won my first challenge and I was just getting used to this awesome community. So, I read the description and started to do a proof of concept with the Google Prediction API. This API is awesome, as it works by “learning” samples and then predicting a result for given data. For a guy that has the sad record of failing an easy statistics course five times in college, this is gold! It can be really abstract, really complex statistics “under the hood” and let the developers focus on the functionality required for the app.

Diving In:
The challenge was to get an accurate estimate for the migration time of the GAMME tool, an app developed by Google to migrate from Outlook servers to Gmail, up in the clouds. The logs were provided with some sample data on the migration.

I started with a proof of concept with the prediction API, they provide some cool examples that can be used for free. After crafting a CSV from the challenge assets, I got successful results, so I dove into the coding!

I went with Google App Engine for the development, because it provides seamless access to the Google APIs, and this project is using three, Cloud Storage, Google Drive and of course, the Prediction API. Looking into the past, I think I wouldn’t chose it again, because of some constraints of the project, like the 60 second limit for requests, or the limited java classes usage allowed in the Java Whitelist, but I still think it is a great platform to develop in.

For the frontend I used Twitter Bootstrap, which I am a huge fan of (who isn’t?), for its general awesomeness. I imagine this app being run by a consultant in front of the client prior to a migration, perhaps on a tablet, so its responsive design makes it also cool to run on a portable device. This is how it looks on iPad:

Nostra Gamme:
The initial task was simply to upload the CSV via upload to Google Drive, then trigger the prediction with the data provided. It should be able to predict the time spent migrating, depending on the server and thread numbers, for the number of emails, contacts and calendars given. It will also predict the other way around, it will give the number of servers required to migrate in a determinate time span. To achieve this, two groups of prediction models were created from the original data, as the output for the prediction changes. I thought it would be cool to visually display the data, so I added jqPlot, which is great, although a little tricky sometimes. Here is a screenshot of the first version:

The Great Refinement:
After some positive feedback from the judges, a follow-up contest was launched to make some enhancements. I really enjoyed doing the original one, so this was a possibility to do some enhancements of my own. The requirements were to add client maintenance to manage several models per client. That gave me the possibility to learn (and fight against) the App Engine Datastore. I am a traditional SQL guy, but the Google approach to store data has its advantages, as it is declarative and incredibly fast!

I thought it would be cool to expose some functionality to add the clients, so I added an Ajax-based maintenance, which works with a REST API so it could be called from a client program. This is a screenshot of the frontend:

Fully Entering The Clouds:
I was happy to know there was another challenge open for GAMME prediction, to polish it a bit more. An average calculation was added to compare it with the one given to the prediction API, to have a reference and have a better understanding of the data provided. It also pushed into the clouds the client program extracting the logs from GAMME, which rounded it up to be a fully cloud development! Here is a screenshot of the prediction for the final app:

I would like to thank all Cloudspokes team for the feedback during the development. You rock guys! Keep these cool challenges coming!

Install MySQL jdbc driver on Oracle Weblogic

Create maven multi-module project using eclipse

4 Replies

It is possible to create a maven multi-module project to support special situations we will face when developing with maven. A typical situation would be when we develop a web app and a desktop app, both depending in the same backend logic, like the database access layer. If we are using eclipse as an IDE, this could be a bit complicated to configure, this tutorial shows how to do it.

For this tutorial we have used

Eclipse indigo SR2 with m2e plugin
Maven 3.0.4

But it should work with any version of eclipse and maven.

First we will create the parent project using mvn tool. We browse to the folder we want to create our project in, and then issue:

mvn archetype:create -DgroupId=multi.module.eclipse -DartifactId=multi_module_project

After this, maven will create the pom and the src folders from the basic archetype

Now, we are going to edit the pom.xml, and change the packaging to pom, to indicate this is a parent project:


<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
  http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>multi.module.eclipse </groupId>
  <artifactId>multi_module_project</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>pom</packaging>
  <name>multi_module_project</name>
  <url>http://maven.apache.org</url>
</project>

Now we will create three submodules, one for web, one for desktop, and one common:

cd multi_module_project
mvn archetype:create -DgroupId=multi.module.project -DartifactId=web
mvn archetype:create -DgroupId=multi.module.project -DartifactId=common
mvn archetype:create -DgroupId=multi.module.project -DartifactId=main

Now, if we open the parent POM.xml, we will see that the three modules have been automatically added:

<modules>
  <module>web</module>
  <module>common</module>
  <module>main</module>
</modules>

And that the parent has been added automatically to the child POMs:

<parent>
  <groupId>guide.ide.eclipse</groupId>
  <artifactId>multi_module_project</artifactId>
  <version>1.0-SNAPSHOT</version>
</parent>

Now, we will add the common dependency on the web and desktop projects:

<dependency>
  <groupId>multi.module.project</groupId>
  <artifactId>common</artifactId>
  <version>1.0-SNAPSHOT</version>
  </dependency>

We want the desktop project to output a jar, so we set the packaging to jar:

<packaging>jar</packaging>

Now we will set the web package to a war, so we go to its pom.xml

<packaging>war</packaging>

After that, our multi-module project is ready. So we open eclipse and open the import menu, selecting existing maven projects

We go to the parent directory and then click next, the parent and the three subprojects should appear:

Now we check they have been imported correctly:

You should have the complete directory layout like this:

This configuration will resolve the dependencies with common through the workspace. This is handy because we will not have to perform a maven install each time we change it. To disable this feature, just click on disable workspace resolution on the maven menu:

Please note that we will have to do a Run as -> Maven install each time we change the common project if we disable the workspace resolution. That would push the project to the local maven repository to be resolved after by the other child build.

When we change any pom.xml, we should do a maven update:

And update the three projects configuration and the parent, to be sure that all changes are updated.

Now, when we perform a maven build in the parent, it will trigger all three builds of the children, so we can automate even more the build processes.

Starting with Google Prediction API

1 Reply

Google offers a ton of cool APIs, but one of the coolest is Google Prediction API. It works as some sort of crystal ball with data. You only have to provide sample data into the service, and then it will “predict” future results for new data. The possibilities with this are endless, we can predict future sales according to previous ones, or detect patterns in texts, to see the mood of the writer.

However, I found it a little difficult the process of starting a new project with it, that is why I am doing a kickstart guide to configure a project from scratch. This guide will cover the steps to set up prediction API to be ready to use with a Google App Engine Project. In another guide I will cover how to develop an app to access the prediction API.

Requirements:

A Gmail account (obviously)
Sign up as a google developer in Google Developers Webpage

Create project on Google App Engine

Go to My applications on App Engine control panel, and click create.

Google App Engine menu

This will show the new application creation wizard. Fill in the gaps with your app name and view permissions.

New application wizard

Once the app is created, go to the administration panel, and copy the Service Account Name, because we will need it later.

Detail of app administration

Congratulations! Your app is ready to go! Now let’s give it the necessary permissions.

Enable APIs

Go to Google API console and create a new project.

Screenshot of the console, after creating the project

Using prediction API is free, but we need Google Cloud Storage space to put the data for the training. At the moment, Google offers 5GB for free, but we must have billing enabled to use it. So, we go to the billing option and enable billing.

Enable billing screen

This will require a credit card, but don’t worry, you won’t be charged anything if you don’t pass the courtesy limits. We will limit the calls for courtesy limits later.

Once billing is enabled, enable the following apis from Services menu:

Google Cloud Storage
Google prediction API

Then go to Team menu, and add the App engine service account we copied before as a team member. This will allow the app to access cloud storage.

App added as a team member

This is optional, but I find it useful. Once you pass the courtesy limit, you will be charged for the API usage, if we are only test driving it, we don’t want to pass the courtesy limit. Let’s set the billable limit match the courtesy limit (100 calls / day) so we won’t be charged. This can be changed later when we need more API calls.

Go to quotas, and click on set billable limit:

Setting the billable limit

Now go to the Cloud Storage Console, where we will create a bucket to store our app files. Note also this name for later. In my case it will be gammeprediction

Bucket creation

Now, we have several ways to authenticate the app to use google APIs, in this example I will use the combination service account + key, because we won’t be accessing personal user data. Feel free to use others.

Downloading the key

We will need a key (a .p12 file) and the service account name for our application to work. Go to API access and create a service account. Click on Generate new key.

Downloading the key

When we click on download private key, a .p12 file with our key will be generated. The default password is notasecret, this can be changed from the console, but for now we will keep it as this. After downloading it, note the Email address value, because we will use it as the service account. This will be a combination of alphanumeric characters, followed by @developer.gserviceaccount.com.

So, this is all we need for now, after the whole process we must have:

.p12 file with our key
Service account email
Bucket with permissions on Cloud Storage

That’s it for now! Check the continuation of this tutorial, how to create a google prediction api app in java

Code Pioneer

Discovering things in a hellowordly manner – Eugenio Cuevas

Hitchkhiker’s guide to Topcoder, a Cloudspoker’s view

Register fast!

Think before registering

Make forums your friends

Understand different phases

Pay attention to the details

The ninja guide to hackatons

Sit, relax, think

Start small, grow big

Start with a supporting point

Do not reinvent the wheel, build the coolest car

There is always hope

Divide and conquer

AngularJs log in with LinkedIn

Predicting Migration Times With Google Prediction API

Install MySQL jdbc driver on Oracle Weblogic

Create maven multi-module project using eclipse

Starting with Google Prediction API