Tests for ProcessWire core

[nik]

As briefly discussed earlier with Ryan (http://processwire.com/talk/topic/1560-searching-multiple-fieldtypes/page-2#entry23307), I tried to build some sort of test suite for ProcessWire. The tests aim for selector consistency (in-memory and db) and of course making them free of bugs - once and for all.

The beginning of the test suite with instructions can be found from https://github.com/niklaka/ProcessWireTests . I wouldn't be surprised if I had to go and restructure the whole thing sooner than later (more than once even), but that'll do for now. Next I'll focus on writing a basic set of tests to cover at least the most common selectors (as there's currently only a few of the most simple ones there).

This testing stuff if somewhat new to me and I've been reading lately a little something on the subject. Hope I got at least something right . Having said that, any help (or constructive criticism) from you my dear friends is highly appreciated. Especially if you'd happen to have any/some experience on testing - but I'll take kind words from anyone .

I decided to try where I'm able to get with PHPUnit and so far it looks like it would be very much possible to build something useful with it (no, I'm not exactly surprised as it sure is a proven tool for such a thing). I did take Ryan's idea of using Skyscrapers profile as a base for testing (you're not making changes to it are you Ryan ), but there may be need for some other approach as well, even another Skyscrapers based one.

Well, I'll go and write a a couple of more tests now.

(PS: If some admin feels this post fits some other area better, please do move it - I wasn't sure where to put it myself.)

[apeisa]

One word: super brilliant!

[ryan]

Thanks Nik, this is fantastic! I'm already using and benefitting from it here. Just fixed two PW bugs, as you posted on the GitHub issues report. It's so nice to be able to run this to locate bugs and inconsistencies, and even better to run again after fixing them (to make sure some new problem wasn't introduced). Thanks again for your great work on this. 

[ryan]

Btw, for anyone else trying to install Pear/PHPUnit on the latest MAMP (and possibly other recent versions), Nik's install instructions don't seem to work, at least they didn't for me. I think it's because MAMP has it's own [already installed] copy of Pear separate from OS X's copy. The very last entry on this page at Stack Exchange worked for me. It's the one answered Oct 9 '12 by Allen Hurff. 

One word: super brilliant!

Failed asserting that actual size "two words" matches expected size "one word".

[nik]

Glad you like it .

Do you prefer separate issues on the GitHub for each suspected inconsistency / bug? I guess it's better to have them as issues in case something that's not going to get fixed right away shows up. But I'm ok with sending you an email or whatever as well if some test is failing.

The test suite is quite ok for the moment (after few rounds of refactoring) and I will be adding more tests whenever I've got some spare time. There are things I know will have to be changed to test some aspects, but I think they can wait a bit for now.

Btw, for anyone else trying to install Pear/PHPUnit on the latest MAMP (and possibly other recent versions), Nik's install instructions don't seem to work, at least they didn't for me.

Hehe, I'm using MAMP myself and that's what I did on two different Macs even . Well, maybe I'll just link to a couple of different install instructions found on the web then. Good to know, thanks Ryan.

[ryan]

Do you prefer separate issues on the GitHub for each suspected inconsistency / bug? I guess it's better to have them as issues in case something that's not going to get fixed right away shows up. But I'm ok with sending you an email or whatever as well if some test is failing.

Assuming this is also easiest for you, GitHub issues are great. I prefer this because: 1) I can reply to the emails it sends me and it goes in the thread; 2) I can reference the issue # from a commit and it goes in the thread; 3) It notifies you when I fix it; 4) It's impossible to miss or get lost--it stays there till I fix it. Whereas, in the forum it is easy for me to miss or lose things just because there is so much here. 

The test suite is quite ok for the moment (after few rounds of refactoring) and I will be adding more tests whenever I've got some spare time. There are things I know will have to be changed to test some aspects, but I think they can wait a bit for now.

I need to learn how to use PHPUnit so that I can help with this. I looked through the code, but am not yet grasping how it works. I'm not totally clear what PHPUnit does that couldn't be done with a couple small functions of our own, but I hope to be over my ignorance on that soon. I will read/study more here and get up to speed so I can participate. 

[nik]

Assuming this is also easiest for you, GitHub issues are great. I prefer this because: 1) I can reply to the emails it sends me and it goes in the thread; 2) I can reference the issue # from a commit and it goes in the thread; 3) It notifies you when I fix it; 4) It's impossible to miss or get lost--it stays there till I fix it. Whereas, in the forum it is easy for me to miss or lose things just because there is so much here. 

I need to learn how to use PHPUnit so that I can help with this. I looked through the code, but am not yet grasping how it works. I'm not totally clear what PHPUnit does that couldn't be done with a couple small functions of our own, but I hope to be over my ignorance on that soon. I will read/study more here and get up to speed so I can participate. 

GitHub it is then. It's of course the Right Way to do it as well.

I think there isn't too much in PHPUnit that couldn't be done pretty easily and more straightforward even, at least at this stage. However, in my opinion there are a few advantages in using PHPUnit instead of a home-grown alternative: 1) It's production ready, so no need to re-invent the wheel; 2) It's a known tool, which could give the project some extra credibility; 3) There are some integration/extension possibilities there already that just might come in handy, for example code coverage analysis (with Xdebug) and Selenium integration.

And maybe I should add some more personal ones: 4) It would have taken me ages to come up with a script decent enough; 5) I saw an opportunity to learn something that could be useful in the future. And so on .

But, just to be clear, if there comes anything in our way that would stall the progress of testing at least the selectors, I know I won't be missing any tool and will be more than happy to choose another alternative in no time, be it home-grown or not.

To write tests using PHPUnit the main thing to know is that any method with its name starting 'test' is considered a test and will be run as such. The tests are located in classes inherited from PHPUnit_Framework_TestCase (or ProcessWireTestCase in this case, which adds a little something there in between). I tried to follow some examples on structuring the tests by creating directory for Selector tests and creating the test case classes there. By the book there should be a test case class for every class that's being tested. But as these aren't really unit tests anyway, I took the freedom to group the tests a bit differently. So right now there are two classes under Selector: BasicOperatorsTest and ContainsOperatorsTest (classnames should end with 'Test') and there could be for example SubfieldTest, SortTest, PagingTest etc. I don't know yet how big each of those would grow so it could have been better to have just one class for starters - especially when with this kind of naming it wouldn't be clear where some more complex tests belong to. But there are other maybe even more logical ways to cut this into pieces, we'll see which direction we're going to take.

There are only two test* methods in both of the classes at the moment: one for testing selectors in database and another for in-memory tests. Both methods use a data provider (see the @dataProvider annotation) that returns an array of arrays of arguments to pass to the test method. So basically all you need to do to add a new test that follows the same pattern, is add another array or arguments for the data provider method to return (description, selector string, array of assertions and an optional skip message to mark the test as skipped). More complicated tests can be added as new test* methods as well.

[ryan]

Sounds great Nik, thanks for this info. I think this is a great platform for the test framework. What do you think are the next steps to take? Let me know what I can do to contribute -- maybe assign a task to me?

[apeisa]

This looks interesting: https://github.com/IcecaveStudios/woodhouse#readme

It would be great to see if latest dev (or any other branch) passes all the tests.

[nik]

Sounds great Nik, thanks for this info. I think this is a great platform for the test framework. What do you think are the next steps to take? Let me know what I can do to contribute -- maybe assign a task to me?

(It's been quite hectic around here the last couple of days - sorry to keep you "waiting").

Well, it seems I lack the capability to split things into pieces more than one person can work on simultaneously.. But let's see. I did some small structural adjustments, again. At the moment I'm thinking of these separate test classes for selectors:

• ValidityTest - to check that invalid selectors are being reported correctly, and valid ones are accepted
• OperatorsTest - to have some basic tests for each of the operators (this is the part that has some content there already)
• SubfieldTest - to test the different kinds of subfield selectors (parent.* and fieldtypes Page, Repeater, MapMarker, Comments + those I'm forgetting atm)
• SortAndLimitTest - to test various scenarios with sort, limit, start and end

I think these would cover most common scenarios at least. Then there are a few more special selectors probably requiring a bit different approach (count, has_parent, include, status, grant_check), especially regarding consistency between in-memory and db queries. And it may be beneficial to test with a set of very complex selectors as well as there just might be something nasty lurking around - at least after some bigger core modifications if not currently. And I think you'd like to be able to do something like that one day.

I'm quite keen on trying to get some test coverage report out of the system pretty soon as well (@apeisa: yeah, that sure looks interesting - have to see if that would suit our needs). And after that extending tests to cover functions in Page(s), Page/WireArray and so on will be my target. But I guess it's better to try and nail them selectors first .

So, you could basically pick any of those test classes outlined above or just wait for me to submit issues to GitHub . SubfieldTest should be pretty straightforward to do based on OperatorsTest - and there's actually a stub just waiting for you in GitHub already. ValidityTest is another one you could write at once, but that should be done a bit differently from OperatorsTest I think (there's a way to handle exceptions in PHPUnit). Anyway, I'm going to continue a bit with the operators and then head to sort & limit, unless you get there first . Sadly it seems I haven't got too much time to invest on this right now - I hope I'm able to get something done during the weekend though.

[Martijn Geerts]

I don't understand much about the whole "Tests for ProcessWire core", but we all benefit from you effort on all things PW. ( not only this test )

a big THANK YOU to you nik. 

[apeisa]

I don't understand much about the whole "Tests for ProcessWire core", but we all benefit from you effort on all things PW. ( not only this test )

This helps greatly the future development. Whenever there are core changes on selectors (optimization, new features etc), these tests can be run and see that everything works like they should.

AFAIK currently only selectors are tested, but this can be taken much further for all parts of the framework.

[ryan]

Thanks Nik -- I will get more familiar with PHPUnit and then work on the SubfieldTest.

This helps greatly the future development. Whenever there are core changes on selectors (optimization, new features etc), these tests can be run and see that everything works like they should.

This is exactly right. 

[netcarver]

@ryan Are you using this in your tool chain? If so, could you update the repo with any additional tests you've added, like the subfield test you mentioned above? Thanks.