Faster CI - Our Journey To Halving Our Test Runtimes

One of my biggest challenges for the past few weeks was to get our tests to complete within 10 minutes.

That doesn’t sound so bad, does it? There’s just one tiny detail which would make this journey all the more exciting: Our code takes 25 minutes on average to complete on 25 parallel build servers and It takes about 4 hours if you run it in one.

I made a quick survey with our engineers and we lose roughly around 3 hours per day waiting for CI to finish.

The plan was to selectively run tests and have cucumbers run only on specific branches — epic, develop and master. This means that every time something gets merged to those branches, cucumbers will run after rspecs. And of course, all this will run in parallel on 25 build servers.

How our code tree looks like:

|- master
|- develop
   |- epic/adding-a-thing-with-stuff
      |- feature/with-things  
      |- feature/with-stuff
   |- epic/the-thing-you-do
      |- feature/bla-bla-bla

We didn’t have a way to execute this plan with our current CI solution at that time so we had to find alternatives. We tried quite a few but most of them were either had limited features or were just simply too hard to get started.

Enter Codeship. It took me no more than 3 minutes to get my first build up and running…and of course, failing. Getting started with their ParallelCI wasn’t that hard either. How you configure your builds opened up a ton of possibilities including selective test runs.

To make our tests parallelize (is that even a word?) I had to use this neat little gem called parallel_tests.

Getting started with the gem is simple. You can execute a group of rspecs like so:

bundle exec parallel_test spec/ —verbose -n $TEST_GROUPS —only-group $TEST_GROUP —group-by $TEST_GROUP_BY —type rspec

$TEST_GROUP is an ENV variable that I set on each pipeline. Basically just the number of the pipeline. $TEST_GROUPS is the total number of parallel pipelines. This is 25 for us. $TEST_GROUP_BY is how we group our tests and that would be runtime for us.

That is one other thing you can do with parallel_tests is group by runtime information. Using this grouping took 2 minutes off of our test runtimes. Getting logs from parallel pipelines was a little bit tricky so I had to do a little scripting magic to gather logs from multiple pipelines and concatenating them back together.

It’s simple really, in the setup section I have two scripts: one to upload each log slice to S3 and the other to download the logs and put them together.

Links for reference, apologies for the terrible naming: - Concatenate - Upload

Now that the runtime logs are covered, it’s time to make the tests work. I have this bash script in our code:


bundle exec parallel_test spec/ —verbose -n $TEST_GROUPS —only-group $TEST_GROUP —group-by $TEST_GROUP_BY —type rspec

if [[ “$CI_BRANCH” =~ .*”$BRANCH_FOR_CUKES”.* ]] || [[ “$CI_BRANCH” =~ .*”master”.* ]] || [[ “$CI_BRANCH” =~ .*”develop”.* ]]
  echo “Running Cucumber…”
  bundle exec parallel_cucumber features/ —verbose -n $TEST_GROUPS —only-group $TEST_GROUP —group-by $TEST_GROUP_BY —type cucumber || true
  echo “Not running Cucumber”

Conveniently, Codeship has a bunch of ENV variables set inside test instances and one of those is $CI_BRANCH. I just simply check if the current branch running in CI matches either master, develop or what I configured in the settings, it will run Cucumbers after RSpecs.

Finally, this is what our setup and pipeline configuration looks like:

pipeline setup image

This entire approach got our RSpec builds down to an average of 6 minutes and our entire suite to 13 minutes from 25 minutes.

Huge props to the Codeship team for helping out with the transition and thanks for the swag! :)

pirate booty arr

Shady Business

It has been a long day for Rachel. As she was removing her coat, her shoes and her gloves, she paused for a moment and thought about how much she hated that diner along with everyone in it. She just let out a deep sigh. She just got home in time for supper. Tonight is the night. She’ll do anything just to get by. She’ll do anything just to get her fix.

It has been five years since the outbreak. The city and the people have never been the same after that. People have been quite... distant with each other.

The government has implemented a law banning prolonged physical contact between humans. This includes hugging, holding hands and all other forms of human intimacy. It was believed that the epidemic spread because of the human touch.

The streets sure are cold this time of the night, Rachel thought to herself as she’s walking on the pavement three blocks away from home, occasionally looking behind to see if someone’s following her. As she reaches the corner, she pulls out a piece of torn paper from her coat. 6th street, the handwriting says. She takes a deep breath and a swift look to the empty street before she enters the dark alley.

These meeting places are getting harder to find, she said to herself. These “dealers” are always on the run from the police, they have to change their location twice a week. Rachel just shrugs it off and thinks that it is worth it. She needs it.

She reaches the end of the alley with a door slightly ajar. She gives it a slight nudge and she’s greeted with a nice, warm feeling, reminding her of fresh toast and a hot chocolate on a Sunday morning. A stranger comes up to her and greets her with a big hug. She was taken aback for a moment but the surprise on her face was overtaken by her smile.

These intimate conventions happen right under the noses of those in power. It’s human nature to long for the touch and the warmth of another and people are going to do whatever it takes, even if it means breaking the law.

The stranger holds her hand and pulls her to a bigger room. Here she sees people from all walks of life. She doesn’t know any of them but she surely does recognize some familiar faces. Rachel has been going to these small gatherings for three weeks now. The room was loud and happy. She was happy. She was finally home.

Time was moving too fast she didn’t want that moment to end. There surely were a lot of hugs given tonight, Rachel thought to herself with a smile. The walk back home seemed shorter for her. She reaches her apartment door still with a smile on her face. This should get me through another week, she goes thinking as she removes her shoes and her coat. There was no need for gloves this time.

Now a little background:

Writing Prompts is a subreddit where people make stories depending on the situation presented. This is my very first attempt on a prompt.

Source: Physical contact is now illegal, but there are hug dealers and shady hand-holders in the dead of night.

A Fat Guy's Guide To Batanes



Essentials. Wipe the sweat off your sweaty cheeks, read through every item and remember it well.

Deodorant. Pick up a deodorant from your nearest seven eleven. You'll need it more than ever because I shit you not, you'll sweat like a mofo. You need to get those smaller ones because of that retarded rule airports have.

Shirts, lots of em. Remember when I told you about sweating like a motherfucker? A deodorant doesn't stop that. That logic is like thinking a band-aid can stop a bullet wound from draining your blood. Batanes is pretty hot, and you'll be doing a LOT of walking. Depending on your length of stay, the number of shirts can vary. My formula is NUMBER OF DAYS * 1.3.

Sunscreen. Did I tell you about Batanes being really hot? We don't want you getting yourself closer to having skin cancer. I can also say a joke about you being cooked but I think I just did.

Scarf. Remember when I told you you're gonna swe-

I'm gonna stop reminding you unless you have alzheimers or something.

Having a scarf is pretty sweet. You get something to wipe your sweat with, sun protection on your face and bonus points for being fabulous.

Watch your fellow traveller's faces melt with envy because you have that one item they wish they had.

Sunglasses. Don't travel like a plebeian. Get yourself a nice pair of sunnies.

Rubber shoes or a hiking shoe. Stop wearing your fancy boat shoes for once. We don't want your ass slipping when you're walking around the Rolling Hills. Let's face it, us fat guys, we're pretty heavy. We need those shoes to clamp us down and hold us in place.

Obviously, these aren't the only things you'll need when travelling. These are the things that are really easy to miss and forget. As for other stuff, go ask your mom or something.


Moving around Batanes is pretty easy. You can choose from motorbikes, bikes, trikes, jeepneys, vans, and good ol' walking shudder.

Trikes are hard to come by. The trick is to call them to pick you up and take you to your desired location.

One of our favourite drivers here in Basco is Dale (09293410941). He'll be your tour guide and driver all in one. His trike has a far superior ass-to-seat ratio compared to other trikes. Shit, you can probably fit a lounge and a minibar inside.

If you're a cheap bastard and don't mind bending yourself over to sit, this one's for you. But hey, at least you can rest your head in your belly while travelling.

Rates on getting to places can vary. But here are the most common ones that would probably be useful. Rates listed are good for two people.

Sabtang Island

North Batan Tour - ₱1,000.00

South Batan Tour - ₱1,500.00

Trip to Port Ivana - ₱220.00

Sabtang Island Tour - ₱1,300.00

Bikes. Being a fatass doesn't mean you need to act like one. Using bikes to travel around Basco is very practical and cheap! Rent one of those city bikes and always bring a bottle of water handy. Bike rentals are typically around ₱500.00 a day or you can get hourly ones at ₱50.00 per hour.

Biking in Batanes

Van rentals are abundant and the true fatass way to go. Complete with AC and huge legroom, it's more than anyone could ask for.

ATV. Just recently, a new shop opened for ATV rentals. And holy fucking shit it was awesome. You can drive around the island for only ₱300.00 an hour or ₱1800.00 for the entire frikkin day.

ATV in Batanes

I cannot stress this enough, you must, MUST, try this at least once. Give them a call at (09985519656/09085012288).


After a half day of walking and sweating, you must feel pretty hungry. Luckily, Batanes has a wide array of restaurants to choose from. Four to be exact. So many!

Pension Ivatan wins my taste buds. They cook pretty good food. You must try their yellow rice! One of the best things about this place is wait for it... They deliver. Game changer right there. Your move, other restaurants.

Their price ranges around ₱150.00 - ₱300.00 per dish. They open at 6am to 1pm. Then 4pmish to 6.

Foodie Score: ★★★★☆

Vatang Grill and Restaurant is quite far from the main town but they do have some good food. Their menu offering gives you a choice of grilled pork, chicken and fish. You can enjoy looking at pictures of celebrities while eating. Isn't that fun?

Their price ranges around ₱200.00 to ₱300.00 per dish.

Foodie Score: ★★★☆☆

Cafe Du Tukon. Holy shit this place is expensive. It's ₱400.00-₱500.00 per dish. The reason for this abhorrent pricing is they are owned by former Secretary Butch Abad's family. Your tax money at work here folks. Oh, and the awesome view. Their turmeric chicken is pretty delicious though.

Cafe Du Tukon

Foodie Score: ★★★☆☆

Octagon Bed & Dine. Meh, although their view of the sunset is pretty sweet. Prices around ₱150.00-₱200.00 per dish.

Foodie Score: ★★☆☆☆

Bonus. Since you've got this far, I'm gonna let you in on a secret. There's a small store just outside the airport called Canteen Batanes. It serves the best nilagang baka in the entire group of islands for only ₱65.00 per serving. Each serving can feed an entire family of four. In fatass terms, just you.

Foodie Score: ★★★★★

Where to Stay

You must feel pretty sleepy after eating all that food. I mean, it's a stressful task. Basco is riddled with lodges and homestays you'll never have a problem looking for a place to stay. That rhymed.

But since this is a fat guy's guide to Batanes, I will be listing places to stay, other than the overrated Fundacion Pacita.

Timetravel Lodge is comfortably located a few frikkin meters from the airport and a few steps to a restaurant to boot. If this doesn't spell fat-friendly, then lern2spel, stoopid.

With two double beds, a TV, a private kitchen and bathroom, you have your own house in the centre of Basco for ₱1500.00 a night. Holy hell, that's kind of affordable!

Give ate Linda a call (09074335610) and send her my regards.

Were you expecting me to list more? I'm not your travel agent.

I swear on every food I eat, Timetravel Lodge is the best place to stay in Basco.

Closing Remarks

You'll probably lose a few pounds in the few days you'll be staying in Batanes. Partly because the food is expensive. You'll most likely gain them back in a few hours back at home anyway.

All in all, experiencing Batanes was one of the best things that ever happened to me. All the blogs and features about it, including this one, doesn't even do it justice. Ivatans are the most friendly and respectful people I've ever met. To me, this contributed a lot to the experience.

You may lose your breath along the way, you may shed a few liters of sweat on a daily basis but goddammit, go out and realise the dream that is Batanes.

Necessary Evils

George beamed on top of the stage to the crowd with a huge smile on his face. Loud cheering, applause and a slow chanting of his name echoed through the sea of shiny black robes with tiny hints of gold. He motioned his hand to stop the crowd's cheer. He is ready to speak now.

"Thank you my fellow graduates! Thank you!"

The crowd went quiet, as if urging him to speak. Everybody loves George, he's smart and kind. He's everyone's friend. He's also this year's valedictorian.

"It has been a long year for us and now we've finally reached the end of our journey."

George went on with his speech talking about hard college life was for him. How he braved through exams and school activities. He went on thanking everyone who was part of his journey in school.

"I would like to thank my schoolmates, my friends. Without them, school wouldn't be as bearable as it is."

The crowd cheered. Everyone was his friend after all.

"I would like to thank my parents for their undying love and support for me. All of this wouldn't have been possible without them."

George paused for a bit. There was a hint of hesitation on his face but he went on.

"and lastly... I would like to thank one of my teachers. The one who told me I wouldn't make it the first semester. The one who told me I will amount to nothing."

The crowd fell silent.

"Well, here I am Mrs. Flannery, here I am! I have proved you wrong. Everyone graduating here with me right now, proved you wrong."

The crowd erupted with cheers. George ended his speech and on that note the ceremonial rites were over. Everyone shook everyone's hands, it was a celebration.

The parking lot was empty except for Mrs. Flannery. Her car keys were slipping from her hands as she's trying to enter her old, beat-up suburban. Her hands were sweaty, probably because nobody shook them.

Mrs. Flannery arrived at her empty apartment aparently a little shaken but still composed. All these years of teaching had made her an emotional rock.

I'm proud of you George. I knew you had it in you. You were just a little lazy. I knew your potential from the very first day of school. You just needed motivation. You needed a push. I'm proud of you. I'm really proud of everyone.

Mrs. Flannery doesn't believe in spoon feeding her students. She wants everyone to work for their achievements and not have their diplomas just being handed out to them. She did everyone a favor on her expense, and it looks like she succeeded.

Mrs. Flannery walks to her desk, and turns on her lamp. She takes a seat and her cat jumps on her lap. She takes out her pen and her notebook. It's time to plan for the next school year.



I can hear the steady beeping of the machines, see the flickering lights and hear the sound of air pumping. It’s like an orchestra of machines playing the symphony of life.

Peter was always a sickly child. He never had the chance to go out and play as much like all the other kids his age do. Normally you would see Peter sitting behind his window looking outside, seemingly lost in his own thoughts.

Peter may have been sheltered, he may have not explored the world as he should have, but boy, his imagination took him to places he can never go on foot. Peter was a creative child. His mind was his playground and his creativity was the limit.

I remember the time he told me about this place he had been to, I just can’t remember what he called it. A place of magic and adventure, a place of magical forests, pirate treasures and fairies. A place where every child wants to go. A place where you can do anything you like, eat anything you want. He even told me he could fly with just a happy thought. A place where time doesn’t exist and no one ever grows old. Peter’s stories were so detailed and specific, I swear I almost believed him.

I’ve been taking care of Peter ever since I found him in an empty alley and brought him to the orphanage. I’ve always wondered how a mother can leave a child like that.

Because of Peter’s condition, he really wasn’t able to make a lot of friends. I remember the day when Wendy and her young brothers John and Michael came to the orphanage. You can see the sadness and confusion in their eyes. Their parents died in an accident and they had nowhere else to go, no one in their extended family wanted to take them. Peter was the first one to talk to them and make them feel welcome. They’ve been friends ever since.

Every day Wendy and her brothers would come visit Peter in his room and they would listen to all his stories. I would join in too, of course. They can’t have all the fun all to themselves.

It wasn’t all rainbows and sunshine for us in the orphanage. Every 12th of the month, people, couples, would come in to the orphanage and get kids for adoption. Everyone would be up and ready by 4 o’clock in the morning. Mr. Hook, the oprphanage headmaster, would come and inspect all of us to see if we’re ready for our potential “parents”. While it was easy for young ones to get adopted, it is not the case for us who are older. I guess belonging in a family would never really happen to us. No one wants an older kid, and no one wants a sickly child even more.

There was something weird about the noises these machines are making. I’ve been with Peter for two weeks and this is the first time they’ve been acting this way. I’m nowhere near a doctor, but I could definitely tell something is wrong.

I call the head nurse’s office and tell them about the situation. “We’ll be there right away Miss Bell!”. I notice a hint of panic in her voice and I can hear them calling for the doctor in the background.

The doctor and his group of nurses enter the room in a matter of seconds. They immediately surround Peter and mutter words I can barely understand.

I didn’t notice that tears have been rolling down my face for quite some time. I was too focused on Peter.

The machine gave a steady, high pitched sound.

“We’ve done everything that we can.”

“We’re so sorry.”

I walk over to Peter and it’s almost as if he’s still sleeping. He looks calm and happy. I can see a tiny smile on his face. I give him one last kiss on his cheek. He’s finally there at that place. A place where every kid wants to be. A place of magic and adventure, a place of magical forests, pirate treasures and fairies. I remember it now, it was Neverland.

I watched Hook when I was bored and was looking for random channels on the TV. I guess you can say that I was hooked! YEAAAAAAH

“I do belive in fairies! I do, I do, I do!”

Photo taken here