Once upon a time, I was building my First Serious Rails App. I was drawn to Rails in the first place because of automated testing and ActiveRecord; I felt the pain of not using an ORM and spending about a week on every deploy making sure that things were still okay in production. So of course, I tried to write a pretty reasonable suite of tests for the app.
To gloss over some details to protect the innocent, this app was a marketplace: some people owned Things, and some people wanted to place Orders. Only certain Things could fulfill an Order, so of course, there was also a ThingType table that handled different types of Things. Of course, some Types came in multiple Sizes, so there also needed to be a Size Table and a ThingTypeSize table so that a User could own a Thing of a certain Type and a certain Size.
Stating that creating my objects for tests was difficult would be an understatement.
Then I read a blog post about FactoryGirl. Holy crap! This would basically save me. With one simple
Factory(:thing) I could get it to automatically build a valid list of all that other crap that I needed!
So of course, I had to write my spec for a thing:
describe Order do it "generates quotes only from Things that are of the right size" do order = Factory(:order) thing = Factory(:thing, :size => order.size) thing = Factory(:thing) order.quote! order.quote.thing.should == thing end end
This test worked. It also generated around 15 objects, saved them in the database, and queried them back out. I don’t have the code running anymore, but it was like 30-40 queries, and took a second or two to run.
That was one test. I was trying to test a lot, even though I wasn’t good at test first yet, so my suite got to be pretty big. Also, sometimes my factories weren’t the best, so I’d spend a day wondering why certain things would start failing. Turns out I’d defined them slightly wrong, validations started to fail, etc.
This story is one of Ruby groupthink gone awry, basically. Of course, we know that fixtures get complicated. They get complicated because we have these crazy ActiveRecord models, don’t use plain Ruby classes when appropriate, and validations make us make extra objects just to get tests to pass. Then fixtures get out of date. So let’s introduce a pattern!
Of course, since we know that Factories are really useful when things get complicated, let’s make sure to use them from the start, so we don’t have to worry about them later. Everyone started doing this. Here’s how new Rails apps get started:
Woo! Failing test! Super easy. But what about that test time?
Finished in 0.01879 seconds
Now, test time isn’t everything, but a hundredth of a second. Once we hit a hundred tests, we’ll be taking almost two full seconds to run our tests.
What if we just
Finished in 0.00862 seconds
A whole hundredth of a second faster. A hundred tests now take one second rather than two.
Of course, once you add more complicated stuff to your factories, your test time goes up. Add a validation that requires an associated model? Now that test runs twice as slow. You didn’t change the test at all! But it got more expensive.
Now, a few years in, we have these massive, gross, huge, long-running test suites.
Now, I don’t think that test times are the end-all-be-all of everything. I really enjoyed this post that floated through the web the other day. I think that the ‘fast tests movement’ or whatever (which I am/was a part of) was a branding mistake. The real point wasn’t about fast tests. Fast tests are really nice! But that’s not a strong enough argument alone.
The point is that we forgot what testing is supposed to help in the first place.
A big feature of tests is to give you feedback on your code. Tests and code have a symbiotic relationship. Your tests inform your code. If a test is complicated, your code is complicated. Ultimately, because tests are a client of your code, you can see how easy or hard your code’s interfaces are.
So, we have a pain in our interface: our objects need several other ones to exist properly. How do we fix that pain?
The answer that the Rails community has taken for the last few years is ‘sweep it under the rug with factories!’ And that’s why we’re in the state we’re in. One of the reasons that this happened was that FactoryGirl is a pretty damn good implementation of Factories. I do use FactoryGirl (or sometimes Fabrication) in my request specs; once you’re spinning up the whole stack, factories can be really useful. But they’re not useful for actual unit tests. Which I guess is another way to state what I’m saying: we have abandoned unit tests, and now we’re paying the price.
So that’s what it really boils down to: the convenience of factories has set Rails testing strategies and software design back two years.
You live and learn.