Technology At Large

Basically the same as the video I posted a few of months ago, but with awesome animation done by RSAnimation

SQL vs. NOSQL

Let me just start off by saying that this is not a benchmark or a technological comparison between different implementations. Rather, this is a discussion about a discussion.

I’ve been reading a lot of blogs and seeing many presentations about the subject of NOSQL and its effect on the SQL world. Most of the information available out there is interesting and educational, however, more and more I’m seeing extremists (yes, even technologies have those) ranting about the use of one or the other, claiming that one is “dead” or “dying” or just flat out saying that using a technology is “bad”.

This troubles me to some extent, a technology in itself can’t be “bad”, sure it can be misused but it can’t be wrong. That’s why I thought I’d share my own view on the discussion:

SQL is dead

The first and most provocative article came from Eric Lai and declared that the days of the SQLs databases are numbered. It seemed that the article hit a chord with many people, first it announced the creation of the NoSql movement which I think was a blessed initiative.

I know that one of the bigger issues I have with the way many places teach CS is that you are taught to have a “judgmental view” and not to accept technology at face value. However, when you go to learn a class such as database theory they are predominantly about SQL and not the different data models.

Secondly, the author goes on to claim that SQL is past its prime and will eventually be replaced by the NOSQL alternatives. In other words NRDBMS will replace the RDBMS.

Other bloggers riding the wave of popularity quickly joined in the fight claiming, as one person said it, that “SQL will be to our children as COBOL is to us”.

The problem with “popular” views is that more often than not they are disconnected from reality, which is clearly the case here. For one, RDBMS and the SQL language are still the standard for most of today’s applications. Not only that, but the relational model itself provides a very well defined behavior which is what most companies look for in a database.

I’m not saying SQL will be here forever, it may eventually be replaced by a different data model, but whatever that data model will be it would have to rely heavily on some well defined query language which takes its roots from algebra.

NOSQL is just a fad

The other side of the spectrum comes from people who believe that everything which is currently done by NOSQL can be done by an RDBMS, and that eventually advancements in RDBMS software will render NOSQL obsolete. I feel the resistance from the “SQL camp” is far greater than that of the “NOSQL camp” but I guess that is to be expected.

People are always weary of any change especially ones that shake the very foundation of their beliefs. As I said in my side note above, if you are taught database=SQL it is very hard for someone to tell you database=whatever works for you.

Some people attack the claims of NOSQL providers while others choose to attack the basic foundations on which many of them stand, some even try to make fun of them. None are especially informative or meaningful.

 On top of that there are many people who are genuinely concerned about their position in this crazy NOSQL world, such as DBAs and RDBMS specialists.

I know that I’m only speaking my view, but I feel these fears are unjustified. DBAs will always be needed, even more so in companies that use NOSQL alternatives where the challenges of optimization and efficiency are much more complex and usually much more strict.

Saying that NOSQL is a fad is just like burying your head in the sand. Non-Relation alternatives have existed for quite some time and the recent data models such as BigTable, Dynamo and graph-based (I know it isn’t new but the “useful” implementation is) have come here for a reason.

There are many limitations to a relational model, even if theoretically every data model can be represented by the relational model it still doesn’t infer that every data model should be represented by one.

NoSql data models are fulfilling the need that many companies, which were forced to either use an RDBMS or write their own database, have. Companies such as Google, Amazon and Facebook have been using NoSql solutions for years, whereas more and more companies are moving their infrastructure to these data models with every passing day.

Summary

Everybody likes to root for the underdog and I guess that the NOSQL movement was clever to go against SQL when it started, attacking the “big dog” is a good way to attract attention to yourself. That said, I think that NOSQL has gained enough steam to persist on its own right and the time for endless bickering over which is better SQL or NOSQL is over.

There is no “better” or “correct” solution for all problems, whenever an engineering solution is needed it’s up to the engineers to figure out what works for them given all the limitations they have. It will still most likely be SQL as long as people know that it can also be something else…

Testing as your foundation

estimated reading time: 10min.Testing

Before dispensing any advice about design, infrastructure or any work methodology I thought it best to start with the foundations.
Testing is one of the core aspects of programming. I know it’s one of those tasks that no one wants to do and is often considered a nuisance by developers, but pound for pound it is by far the most rewarding.

However, in order for tests to be effective they must protect the developer from making mistakes he might be oblivious to. If you commit a change to the code that causes an error in someone else’s test you need to know about it as soon as possible, the longer you wait the harder it will be for you to find the bug later on or even to fix it.

So what you need is system that is aware of all tests available in your code, can execute all these tests whenever a change to the code is committed (or every few minutes) and in case of a failure in the tests notify all the developers that changed the code.

In other words what you need is continuous testing…

Continuous testing is probably one of the most useful things any application can use. It requires very little time to set up and has far reaching benefits in both development time and quality.

In an attempt to provide you with a real world example of how to apply continuous testing in your project I will take a different approach from my counterparts and address the most difficult aspect of using a continuous testing framework: convincing your coworkers it’s good for them.

For that reason I am not going to give you many code snippets or talk about available technologies, but rather on the human side of the equation starting with…

Your boss

As we all know there is always something really urgent that has to be done right now and can’t possibly be postpones by something as superfluous as a testing server, right?

Wrong! If time and effort are important then continuous testing is paramount. Try explaining the following points:

Efficiency

using continuous testing (CT) allows developers to find bugs as they are introduced to the system. Because of this if a developer accidently breaks some previously existing behavior, he is notified immediately (or fast enough), before many other changes had been done to the code and while the developer still remembers all the changes he made.
The bottom line is, less time is spent on fixing previously working code and your boss gets more bang for his buck.
If he is still not convinced try this…
Assuming you only save 5 min. a day by using CT (from my experience it s significantly more). Multiply that by the amount of developers working on the project and you will get the daily saving of a CT system.
For instance if you are 3 developers then is one month (22 work days) you will save
22*3*5/60 = 5.5 work hours, so even if you spend half a day setting up some basic form of CT your work will repay itself within a month, keep in mind that as your application and team grows in size and complexity so will the time earned per day.

Quality

By testing everything all the time and notifying people about the status of their tests you are preventing code degradation from occurring, there is a significantly smaller chance that your developers will break some backward compatibility or some existing feature if all such features are tested every time a change is introduced.
Furthermore, by continuously providing feedback to the developers about the status of their tests and by combining tool such as code coverage, method complexity and coding style as part of the report, developers are forced to think about testing whenever they write a new piece of code, making writing tests an integral part of the work process and not some deferred after though.

Time To Market (TTM)

The holy grail of management, in my previous work place reducing the TTM was a major thing, mainly because it consistent of a product developed by 4 teams and about 30 developers over the course of months.
When the product was ready for delivery, each team would merge all their changes to the code and then make sure all the tests were passing. Since some usually aren’t passing this requires a difficult operation by which the merge process requires the involvement of many developers and may take several days. Once the tests pass a code freeze is announced and functional tests ensue followed by a round of fixes and then functional tests again.
When this is over the product was given to integration and QA for testing, overall deploying new code would take about 1-2 weeks.
If we were to use CT that time would drop significantly since merging and unit testing the code is free (it is done on every change). Not only that but with some more effort functional tests can also be integrated into a CT environment making them less likely to fail as well, and if that is not enough then consider the fact the since the quality of code and efficiency of your developers had increased the process of fixing bugs has also decreased in size and complexity.
If you feel that that does not apply to you I can tell you that I currently work at a startup company developing a large scale internet application server and since employing CT the deployment process has changes dramatically.
Before CT deployment was a fearful process where we would all take a break from our jobs and publish the new code version to a single server. We would than monitor all the servers to see that nothing was crashing if it was we would revert the change and try to find the bug if not we would package the code and publish it to all the other servers.
We would then continue to monitor these servers and see that they do not crash once all the code has been published. All in all it was a lengthy nerve wrecking process that took hours of work for all people involved.
Now that we have switched to CT code deployment is essentially free, the code is already thoroughly tested and production ready at any stage in development. So pushing the changes to the servers is just a matter of uploading the new code, in fact after you use some good automated functional tests the need for active deployment can all but disappear and you can have your build server deploy the code to the servers on a daily basis (nightly-builds) and thus deploying a new version of your project just becomes the act of deciding which nightly build will be used as the “stable” version.

So now your boss is convinced, great! But still to come is your biggest challenge… your coworkers.

Developers

Getting your coworkers on board is harder than you might initially think. Instead of giving out main bullets like I did before I’ll try to address some the resistance points you might encounter.

Fix all my previous tests? I don’t think so

Some projects already have many tests integrated in the source code, however, since they are not maintained in an automatic incremental way ghost tests from previous builds can exist all over the code.
What happens when you have many old deprecated tests in your code which don’t pass? Once you start the automated tests all the old unmaintained tests will fail and no one wants to start spending all of his time getting some old deprecated tests to pass on top of all the stuff the developer has to do anyway.
Let’s dump the self righteous claim that you should not have failed tests in your project and deal with the actual problem.
You don’t need to get all tests to pass as a first stage of automated testing, instead only get the ones you are running anyway to pass and any new ones as well just comment out any old test you aren’t running on a regular basis.
Remember the goal is not to be as correct as possible but to get people used to the idea that tests are good. The old tests that aren’t passing will be fixed in time when they become important.

I don’t want some machine to notify everyone whenever I mess up

The big fear people have is usually that once some automated testing server is up every bad move you make is broadcasted to all other developers. First of all, it doesn’t have to be that way most automated testing servers allow notifications to be sent only to the people that committed any changes since the last check, and keep in mind that most times that will only be the one person that introduced the bug.
Secondly, once automated tests are up and running and a full test is conducted every few minutes failures will stop being some sort of blame notification like developers are used to and instead will become the development tool they are supposed to be.
A developer will commit his changes and then after a few minutes will get a notification if he introduced any errors so the developer no longer has a need to run a full system test every time he commits a change it will be run for him protecting him from costly mistakes later on.
And finally, since every commit will cause a full test to run you can expect failed builds (execution of tests) to be a thing of the norm and part of virtually any major code change. I can tell you that in my current project the build fails several times a day, it stops being an issue and no one gives it any other thought apart from the developers that introduced the changes.

I don’t want to spend the time to write tests to everything I do

You’re not really spending time think about it as an insurance policy. If someone changes the code in a way that causes your component not to work he will be notified in time. No more trying to find bugs in your code with the bosses breathing down your neck only to find it resulted from someone else’s change.
Anyway, not writing tests is just like sticking your head in the sand. Your code either works as it’s supposed to and in that case your tests will make sure it stays that way or it doesn’t in which case you’d better know about it sooner than later. The more time and code passes from the time of introducing a bug to the time the bug is recognized the harder it will be to find and fix said bug.

Conclusion

Continuous testing is a great tool for any developer in any project. And with so many easy out-of-the-box testing tools readily available it only takes a minimal amount of time and skill to bring up a continuous testing server. I tried to give you the “How-To” guide for people rather than software, the tools themselves aren’t really important but for the sake of being thorough I will add a small post about the testing architecture I use later on.

Please tell me what are some of the resistance points you’ve encountered and how you handled them…

Autonomy, Mastery and Purpose the notion of 21st century management…  brilliant

Taking the plunge – Conclusion

In conclusion I’m sorry to say there are no absolutes or certainties in choosing to move to a startup company.

The goal of this blog post was to give the reader some insight into the most important component in making the decision, your character.

My current view is that experience in both types of companies is invaluable. I would suggest working a couple of years in a large company; teaching oneself organization, time management, networking (with people), integration and presentation skills followed by several years at a Startup improving your engineering and problem solving skills, as well as learning to work in a very dynamic and challenging environment.

In an effort to provide some real help in your decision try and understand the reason you are looking for a change:

  • Are you looking for new challenges or are you not being challenged enough?
    The former can usually be handled within your workplace whereas the latter may not.
  • Do you see yourself starting your own startup company in a couple of years?
    If so remember, the skills and people you meet in a large company can help you significantly when starting your own company. Many successful startups started as a result of large groups of people or entire teams leaving a large company together.
    If this is your goal, make sure you learn what you can from the company before leaving.
  • Are you looking to strike it rich?
    Stop looking. The best startups aren’t headed by stock brokers but rather visionaries. Money is a byproduct of fulfilling your vision.
  • Can you handle the pressure of the bad times?
    Startups are awesome when everything is going well, however, they can be outright depressing when in trouble. The feeling of the noose tightening is never as intense as it is in a very small company.
  • Can you handle the pressure of the good times?
    Can you spend the amount of time required by a startup when business is booming? Keep in mind startups have less resources to shift around to handle work load.

I hope this helped someone with making his decisions, again I risk being redundant by saying that it is a very personal decision and everyone needs to find the reasons that apply to him.

My decision was to leave and join a startup. It is a hard decision especially considering the lack of spare time and the friends I left behind. It was also scary to start anew trying to figure out my place and the dynamics of a new workplace.

That said I have no second thoughts about moving to a startup, I believe the time and experience I acquired in my previous job serve me well in my new one, and I can honestly say that I think today I am a far better engineer than I was a year ago. Not only that but I enjoy my work a great deal more since I now feel a greater connection to the company and the product we produce.

Taking the plunge – Part 3: Time and risk

Time

Well this seems pretty straight forward, how much time does your job take from your life.

I will state the obvious and say that moving to a startup requires you to spend more time at work than a large company requires of you.
Now I know that some might say that working in a large company can be just as time consuming as working in a startup and that they don’t think people at startups work more hours than they do. And while that may be true in principle and the amount of weekly hours spent on the job will be the same, there is still a difference.

When working in a very small company, a single person is usually very hard to replace.
This makes taking time off in a startup much more difficult than in a large company.
Furthermore, if you know you are heading into a time in your life where your time at work will be diminished like finishing your Masters degree or expecting a baby. Large companies can usually accommodate these changes with relative ease, you do less time and the work load splits evenly.

The possibility of slowing down and investing less time at work at specific periods in your life will always be easier in a large company where there are many people who can cover for you.

When you are approaching the decision you need to look ahead at your life and say:
how much time will I be able to invest at the job a year from now?

Risk

By risk I mean the threat of losing your job, this is a very tricky subject.

Large companies provide a security blanket that allows good engineers to keep their job even if their entire team is downsized. Furthermore even if large companies encounter heavy losses and decide to close shop; it is still a very lengthy process that may take months or years to complete.

That said. Small companies have the advantage of close personal relationships between the management and employees which may mean that a person is less likely to be fired even at the expense of some discomfort to the company.

Risk assessment in startups is very hard and not being an expert on the subject I don’t think I can offer a lot of insight into the process of evaluating risk, however, I will say this

Before deciding a startup is the place for you, sit down with the CEO and have a talk. Technology alone doesn’t make successful startups, the people steering the boat need to be visionaries with mental flexibility. The kind of people who aren’t afraid to fail, have faith in what they do and at the same time are capable of changing direction quickly enough if needed.

I think the last point is the most important and most difficult to assess when choosing a startup.

Taking the plunge – Part 2: Personal Development

Personal development

Personal development is key to the choice of job, mainly because the route taken differs greatly between the two.

Startups are usually characterized by fast solution, burning problems and innovative technologies; as such they provide the perfect platform for building up your engineering skills.

I know some jobs at big companies offer very big challenges, but they rarely entail the wide array of skills required for building an entire application from the ground up.

Working in a very small company requires you to think about everything from deployment, integration and customization to security and testing. No aspect is covered by some other team or department it’s all you all the time.

While large companies can produce some great engineers, they will still be by definition limited to a specific area or field. From my perspective if one wants to improve his overall engineering skills little places can offer the kind of rapid learning and development startup companies offer.

While startups excel at engineering their small size and relative lack of experience make them less likely to offer management positions.

If in a large company you have many teams and many more developers. A highly regarded engineer can be rewarded or advanced into the position of team leader or process manager (PMO) relatively fast. In small companies such promotion is unlikely to occur fast, let’s assume you enter a startup company with 5 engineers. Even if successful the company can’t grow at a very high rate without damaging it’s quality and organization, so even if a year later you number 8-10 engineers odds are there will still be no opening for a team leader position.

Generally speaking, from my experience I would have to say that large companies provide the best platform for developing ones managerial skills whereas startups can help one develop very strong engineering skills.

When I made my decision to switch companies it was largely this distinction that affected my judgment, although I knew I could have probably reached some managerial position by staying at the big company for a couple more years. I felt that I need to improve my engineering skills in order to really understand which is the route best suited for me.

Today, a year later, I can say with certainty that no position at my previous work place could have possibly given me the tools and engineering skills I acquired during my time at the startup.

Like everything else this choice is largely affected by who you are and what your goals are…

Taking the plunge – Part 1: Money

Money

Money was never my reason for switching jobs, I know this is highly personal but the differences in pay between the 2 companies were all but negligible (<10%).

Nonetheless I’ll address some of the more minute differences in pay between the two.

Startup companies are small and as such usually allow their employees greater expenses, you can buy your choice of laptop or computer, lunch and coffee are usually paid by the company in full and generally speaking almost all work related expenses are paid in full.

Another thing is that small companies don’t need to try and cut down on employee salaries using some sort of trickery like big companies do, like giving a small base salary and counting the rest as “bonuses” or “expenditure”. This means you won’t get a de-facto pay cut when missing work or going on vacation. In countries where the employer pays social benefits according to your pay it also means that you get benefits according to your actual salary and not some low “base” salary, for instance in my previous job only 80% of my salary was considered for pension.

The other and to some the biggest allure of startup companies are the options you receive from the company, what it basically means is that if someone buys the company you can receive a relative part of the money. I think that if you choose a startup because of the possibility of striking it rich it is a mistake.
Most startups never reach the point of being sold, and even those that do usually take years to get there, and even on those occasions most times you will not have enough options and the sale price will not be that high so as to give you a huge amount of money.

If you are looking to join a startup in order to strike it rich you might be better off keeping your job and buying a lottery ticket every week.

On the other hand big companies have power in numbers such that small startups can never compete with, large companies might have a gym or some other facilities. Also large companies generally tend to have better offers on cars (lease), health care and other services.

Bottom line is that from my own personal experience if you count the added cost for health care, car ownership and various discounts it is very difficult to say which options pays off better at the end of the month. Maybe I’ll have a definitive answer several years from now :)

Taking the plunge

Jump!Working at a startup

I thought I’d start off my blog with the biggest change that happened in my life in the past year which was moving from my previous company to a startup company.

I want my blog to help people make decisions about technology but also about what works best for them, so I guess the best place to start is choosing where you want to work.

I can’t recommend any course of action for any person other than me, however, I can tell you what were my motivations and my reasoning for choosing a startup over a large well known company and hopefully someone might find it useful when faced with the same dilemma.

About a year ago I was working at a large international software company (>3000 employees) and my contract was about to expire. My previous company was pleased with my work thus far and had decided to offer me a generous contract.

Before my contract expired I decided to look around and see what other options I have available to me. In my search I came to a small Startup company that at that point in time had only existed for 2 years and had approximately 10 employees. I was fascinated by the technology they were developing and after passing the interview was also offered a contract.

And so I was faced with the difficult decision of switching from a large stable company where I was fairly appreciated to a new small startup company.

Now a year later I can look back and share some of my experience with people looking to make the same decision.

I tried to give some sort of logical order to my reasoning, I hope it helps