Current/former Textpattern users: What do you think? (TXP5 using PW)

[MarcC]

The Textpattern CMS is being rewritten using a new framework, but running into some roadblocks because the framework is pretty new to the development team, which slows things down, and doing anything frameworky and scratch-built seems to be discouraging potential volunteers from contributing code.

I had the idea that perhaps TXP5 (Textpattern v. 5) could be built upon ProcessWire's API, using ProcessWire as a content management framework.

That means that Textpattern could inherit a bunch of great features from a content management framework rather than a blank-slate application framework. Textpattern would still work in a Textpattern-like way. It would still look like Textpattern. It might even mean that templates are still saved to the database. But PW offers some incredible features that TXP users have been requesting for a long time. Everything from custom taxonomies and fields to multi-language front- and back-ends, to user management and ACLs.

In addition, ProcessWire has terrific documentation and comparatively short learning curve required to use its API.

I know we have some TXP users that hang out here, so I thought I'd ask--what would you think of a new Textpattern that was built on top of ProcessWire?

And ProcessWire users, what do you think of the idea of another CMS building on top of PW's API?

Ryan mentioned that he thinks it's a good fit, and would even be willing to maintain a pure framework version of ProcessWire if needed. That's a very generous offer so I thought at least I'd keep the idea bouncing around a bit.

The latest discussion in the Textpattern community (mostly talking about the pace of development) is happening in on the blog and in the comments.

[WillyC]

long time lover of Textpattern, and now sexy times with ProcessWire this idea.like I do. Seem like a good time for all. Fluent in 2 cms it would make me (maybe), and DrupaloomlaPress very well lame by comparison for definite. Forgive my English you will, but aroused am I at thinking of thes friendships. But use my ProcessWire I still do most times

[ryan]

The way I see it, ProcessWire is designed for this sort of thing as it's a CMS framework. One can pretty easily build a CMS on top of it and that CMS can have a ready-to-go starting point for just about everything you could want at the lower level. Page, field, data management, API, hooks, access control, etc. One never even has to use database queries (though there's a database API var too called $db, if you want it). All of this is driven around an accessible API, and all the complex stuff happens behind the scenes. The front-end Admin that you use in ProcessWire is basically just a web site/application that's built using that API.

ProcessWire is a more flexible framework than most in that it doesn't try to force you into anything. If you don't want to use some part of it, you don't have to. You can put your files where you want to. If you want to use MVC, great. If not, no problem. ProcessWire is not your mom. It's meant to be your friend like jQuery.

If another project picks up ProcessWire as a back-end CMF, I think that only benefits both projects. It's also a good opportunity to actively maintain a separate pure-framework version of ProcessWire, which is something that I've wanted to do, but didn't think folks would be interested. ProcessWire was a framework long before it was a CMS. The framework is already there (by using the include/bootstrap), so the framework version would basically be a blank version with nothing but a root, 404 and trash page (no admin pages).

The only issue I see from the framework perspective is namespace. ProcessWire was originally designed with PHP 5.3 namespaces in mind, so it has some pretty generic class names like "Database". Such class names could very well conflict with another application. So we'd likely rename our classes to be more PW specific, or we'd make PHP 5.3 a requirement and bring namespaces back into the fold.

[apeisa]

I am all in for 5.3 requirement. Not sure what version of php most of the hosting companies are offering at the moment though...

[MarcC]

Ryan, Stef (TXP5 dev) did mention namespaces, and it sounds similar to what you are saying, I think? Here's the part where it came up:

Keep in mind these things that we would like to achieve:

* unlimited cats / taxonomy / custom fields

* a more normalised DB structure

* support for DBs other than just MySQL

* more installation options, e.g. choosing a default theme package or pre-packaged baseline to install

* namespacing so we can integrate more easily with other systems (e.g. having a global called “DB” is silly. It should at least be TXP_DB, but a proper namespace for the project would be preferable).

* a more secure tag parser (secondpass: Gocom!) [Note: Gocom is a TXP dev who's interested in security -Marc]

* better security for plugins, e.g. forcing variable types in core functions like lAtts() so plugins don’t have to do it, which will help reduce runtime errors and reduce opening unwitting security holes in plugins

* prevent any old PHP function being used as a tag (rah_function notwithstanding: that’s extended functionality which is acceptable, because it’s caveat utilitor; but it shouldn’t be default behaviour in the core, imo)

* better prefs handling

* simpler admin side interface building (at the moment it’s a real chore to build a screen under extensions — some API calls to simplify things (think: admin-side ‘tags’) would be a real help)

* improved user management API — remove reliance on flat file for privs/groups

* a more AJAXy workflow where it makes sense

* improved image/file support and workflow

That's from the comment section, here.

[vikingkarwur]

I loved the idea, Marc

[Adam Kiss]

Has anyone yet broke this to TXP Team? Do we know what's their opinion?

[MarcC]

adamkiss, I originally emailed the idea to Stef Dawson. He said the idea "actually has merit" but I haven't heard anything since. It's barely been a week since that email, but I think there's a good chance that the current direction (using spark/plug) would be resumed. He's put a year into it already and has some documentation written and a start on a small portion of the TXP5 admin panel (a single tab, but still).

Also, Stef is the only active TXP5 developer at this time.

From the other devs (either plugin devs or TXP4 development) mostly there is a strange reluctance to comment in depth on the entire issue of TXP5 development. I get the feeling that there is a lot of hesitance to build on a framework like spark/plug, but again nobody has really gone into detail about their feelings for what should be changed with TXP5 development when those complaints come up. That could be because Stef is doing all the work.

IMO there are a lot of PHP developers in the world who are sick of hearing about frameworks, period. And for me it's a bit awkward to see ProcessWire confused with a coding framework, just because it's called a "content management framework." Sometimes I wish we could just call it "ProcessWire Developer's Friend CMS Builder 2013--Keep Coding the way you Always Have" or something like that.

I've also heard from former TXP people who are now using ProcessWire as a CMS--positive comments, of course, but I'd imagine that to a TXP developer that could really push the wrong button. Kind of a touchy issue. Touchy because TXP5 development has been slow, I think.

[ryan]

I've always held Textpattern in high regard, so it makes my day just to hear that the developer thinks the idea has merit. I wouldn't expect them to pursue the idea, especially given the time they've already invested. There are probably larger considerations than it appears on the surface, especially for someone that authors a CMS.

I also don't think it solves everything they are trying to do. For instance, PW already has a strong hook system in place, whereas their need is to support an existing one and maintain compatibility with it. Their list also indicated one of the goals as database independence, and PW is dependent on MySQL and certain aspects like indexes (fulltext indexes for example, which are far from universal).

On the other hand, I do think PW as a framework would solve MOST of the things they are wanting to do, just not all of them. But that's only a surface evaluation and there may be much more to it when considered in the context of the TXP system, history, requirements, code as a whole. That's not something that I think any of us can evaluate. And especially not me, since I don't have much experience with TXP. But I think the best thing I can do is just express support and enthusiasm for the idea should it ever become a consideration. Lots of respect to Textpattern as a long running quality platform and group of folks that knows how to build a great CMS.

[MarcC]

Excellent comments, Ryan. I wondered myself to what extent it can be database independent--wondering if it would have to be a short list of supported databases or if it just needs to be open for extensions in that area like some frameworks are. I'm not really sure where that feature request came from, though, and do worry a tiny bit about the scope of TXP5 if that's going to take a lot of work. But Stef is a really good release-early-and-often-type guy, and I can tell he's champing at the bit to get something out. He asked me today if I've had a chance to try out the spark/plug setup yet, so I'll probably do so.

Really interested now to see what kind of project might come by and pick up ProcessWire as the API/framework for a public, open-source CMS. I really think it could be a great platform for getting into advanced publishing UI, workflow polishing, etc. More lipstick and eyeliner than girders. Or something.

[ryan]

I wondered myself to what extent it can be database independent--wondering if it would have to be a short list of supported databases or if it just needs to be open for extensions in that area like some frameworks are.

ProcessWire's selector operators *= and ~= rely on fulltext indexes, and I consider these crucial, which why ProcessWire is build around MySQL (MyISAM). Otherwise we'd be using MySQL INNODB and supporting other databases. The only other databases that one comes across regularly in commercial hosting environments are PostgreSQL and SQLite, neither of which support fulltext indexing, last I checked. So it's not a matter of wanting to support other databases, so much as it is a matter of what we need to do isn't easily done on other databases we're likely to encounter.

Really interested now to see what kind of project might come by and pick up ProcessWire as the API/framework for a public, open-source CMS.

Don't forget ProcessWire is a public open-source CMS, so it's what we're already doing and will continue doing. I stay in business by building sites for my clients with the CMS and they mostly don't know or care about the CMF aspect. But as a web designer/developer, the CMF aspect is the one that impacts my building and maintenance work the most. And it impacts my clients bottom line because it costs a whole lot less to build something complex in PW than it does another CMS (at least those I've used). I'm too impatient and don't think I could be a functioning web developer if I had to build sites around tools like Drupal, WordPress or even EE (and Joomla not even a consideration). There's a lot to like about some of those tools, but just glad I don't have to use them for building new sites.

On the other hand, I've always been intrigued by Textpattern, its tag language and of course Textile which came from it. While I've not developed a full site in it yet, I can tell that it's got some real intelligence behind it and it feels a lot more cohesive to me than the other big platforms. That's why I'm enthusiastic even about the idea of it with a ProcessWire framework behind it and would see it as a mutual benefit to both projects. While I think them adopting ProcessWire for their backend is unlikely (given what they've already invested), I think there's still much we can learn from Textpattern. The fact that we've had so many Textpattern users take a strong interest in ProcessWire has likewise made me interested in the Textpattern project.

[MarcC]

ProcessWire's selector operators *= and ~= rely on fulltext indexes, and I consider these crucial,

I do too. Am I correct in thinking that supporting multiple database systems is a significant limitation in terms of the CMS's database tools & feature set?

Don't forget ProcessWire is a public open-source CMS, so it's what we're already doing and will continue doing.

Of course. And it's a phenomenal CMS. I'm interested in seeing what sort of other projects might use the framework for their own CMS, with different results, or a slightly different philosophy, etc. (who knows?)

and don't think I could be a functioning web developer if I had to build sites around tools like Drupal, WordPress or even EE

This really hit me during my last PW project. I compared development times and feature set with similar websites in other enterprise CMS systems and hurriedly put away my calculator before retroactive depression set in. Ha.

I can tell that it's got some real intelligence behind it and it feels a lot more cohesive to me than the other big platforms

See, that's what I like about it, too. It may be a horse-and-buggy setup compared to some, but if so it's an extremely nice one. I normally find it difficult to keep dogfooding the old stuff once the new dogfood arrives, but I haven't had that experience with my Textpattern installs. I keep them running and enjoy them for what they are.

While I think them adopting ProcessWire for their backend is unlikely (given what they've already invested),

Yep, seems unlikely at this point...but who knows where things will go. And for all I know, it might not be a good fit in some way I just don't understand. I have complete trust in Stef's ability to really get things moving and break through issues that are holding things up.

The fact that we've had so many Textpattern users take a strong interest in ProcessWire has likewise made me interested in the Textpattern project.

It's good to hear that!

[futuresplash]

@ryan said above:

"(...) PostgreSQL and SQLite, neither of which support fulltext indexing, last I checked."

Well, SQLite does support fulltext searching and indexing!

Please see:

http://sqlite.org/fts3.html

Could SQLite be, perhaps, a good candidate for an alternative SQL backend for PW ?

Regards

[gcaplan]

Hi Ryan

ProcessWire's selector operators *= and ~= rely on fulltext indexes, and I consider these crucial, which why ProcessWire is build around MySQL (MyISAM). Otherwise we'd be using MySQL INNODB and supporting other databases. The only other databases that one comes across regularly in commercial hosting environments are PostgreSQL and SQLite, neither of which support fulltext indexing, last I checked. So it's not a matter of wanting to support other databases, so much as it is a matter of what we need to do isn't easily done on other databases we're likely to encounter.

Hmm - as a Postgres devotee perhaps I could mention that we've had full text indexing since the early Triassic:

http://www.postgresq...arch-intro.html

Much more sophisticated that MyIsam too, with lexical parsing to handle plurals etc, stopwords, synonym dictionaries and highly optimised indexing. And of course you get all the other Postgres goodies such as intelligent engineering, bomb-proof data integrity (haven't lost a byte of data in 10 years), and a sensible licence.

<rant>I'm having to use MySQL for a current project, and I've been dismayed by what I've found. This is the 21st century, and they are openly admitting that their query parser is so dumb that they still haven't figured out a way to offer functioning sub-queries! They're expected "some time soon". I'm sure your clients would appreciate the option of using a database built by a community who would never dream of knowingly releasing an upgrade with 300 known crashing and critical bugs: http://monty-says.bl...1-released.html :- </rant>

[ryan]

Sounds like much has changed since I originally did DB research before building ProcessWire. Back then, MySQL MyISAM was the only fulltext game in town (short of using something like Lucene). This wasn't that long ago actually, but it's good to hear that this has changed.

I'm sure there is lots to like about PostgreSQL and I like everything I've read. I would love to support it in a future version of ProcessWire. I'm sure that it has many great things relative to others, like you've stated. But I also think that most web developers' reality is not one where the use of one database over another matters very much. I've used MySQL every day for the last 12 years and have never had a concerning situation or ever lost data in any form. MySQL has been one of the most reliable tools I've dealt with, and is one of the things I've been able to depend on most among all the tools I use. So my point is that these factors may matter to someone that works a lot with databases on their own, but they may not weigh so heavily to the larger web developer audience. Still, I'm very interested and supportive of PostgreSQL support in ProcessWire. The only reason I didn't built in support in the beginning is due to the lack of fulltext index support at the time.

[netcarver]

Hi Folks,

Very informative topic, answers several questions I had about the DB choices in pw.

I too would like to see pw adopt a more flexible attitude to backend DB engine and even sub-engine as I have found InnoDB to be a more reliable storage engine than MyISAM in MySQL. I see that InnoDB, too, has recently gained fulltext index support so I hope whatever DB extensions make it into pw also include a MySQL+InnoDB option.

Further, and keeping with the Textpattern theme that kicked off the topic, there seems to have been a recent trend amoung some Textpattern users to move Textpattern's presentation elements (pages/forms/styles) out of the DB and onto the server filesystem (at least while sites are under development) in order to allow them to be placed under version control. I do this myself for some sites I develop in Textpattern and there is a useful plugin that facilitates this.

Is the same possible in pw?

[Adam Kiss]

Hey netcarver, regarding versioning presentation elements - since ProcessWire deals with raw data and makes no assumption what you'll do with them code-wise, this part is totally up to you - if you wish, you can store HTML in the database (as I did on multiple occasions, where it made sense to store data with basic HTML around it), or you can store 'pure' text, checkboxes, textareas in db and have full code in your template files - so this is totally up to you.

And regarding database, I'll leave this to someone more knowledgable.

[ryan]

Great to hear about the addition of fulltext indexes to InnoDB! Once MySQL 5.6 is more widespread in web hosting, I look forward to testing out and comparing to MyISAM in ProcessWire. Though my understanding is that InnoDB is technically superior in many ways. If that also translates to PW performance equal (or better than) MyISAM, then it would make sense to use that where appropriate and possible.

Fulltext indexes really were our own MySQL/MyISAM dependency, so with PostgreSQL, MySQL/InnoDB and even SQLite now offering support, it will be possible to get PW going in more database platforms without too much effort.

[netcarver]

@Ryan,

Out of curiosity, is there a set of unit tests/fixtures for pw? Something that folks could take and use to ensure stuff they were working wasn't breaking anything? I didn't spot anything during a brief look at the pw directory structure.

[netcarver]

Hey netcarver, regarding versioning presentation elements - since ProcessWire deals with raw data and makes no assumption what you'll do with them code-wise, this part is totally up to you - if you wish, you can store HTML in the database (as I did on multiple occasions, where it made sense to store data with basic HTML around it), or you can store 'pure' text, checkboxes, textareas in db and have full code in your template files - so this is totally up to you.

And regarding database, I'll leave this to someone more knowledgable.

Hi adamkiss,

Ok, so could I just version control all the files in the /site directory? I guess I could, that would kind of make sense.

[ryan]

Out of curiosity, is there a set of unit tests/fixtures for pw? Something that folks could take and use to ensure stuff they were working wasn't breaking anything? I didn't spot anything during a brief look at the pw directory structure.

Not at present, but definitely interested in doing that for the future. It's just a matter of time/resources and putting it towards the stuff that matters most in the short term. Longer term it would be really nice to have something like this, especially as PW gets used more and more as a framework.

Ok, so could I just version control all the files in the /site directory? I guess I could, that would kind of make sense.

I would version control your /site/templates/ and /site/modules/, but not your /site/assets/ as they are tied to DB entries. Though if you combine it with a DB export for each commit, then that would solve it. But this is something we'll have built-in version control for, further down the road.

[Adam Kiss]

But this is something we'll have built-in version control for, further down the road.

big YES for db version control.

[netcarver]

Not at present, but definitely interested in doing that for the future. It's just a matter of time/resources and putting it towards the stuff that matters most in the short term. Longer term it would be really nice to have something like this, especially as PW gets used more and more as a framework.

Ok, look forward to that. At least I'd have some confidence that my tinkering wasn't breaking things -- or an early warning if it were.

I would version control your /site/templates/ and /site/modules/, but not your /site/assets/ as they are tied to DB entries. Though if you combine it with a DB export for each commit, then that would solve it. But this is something we'll have built-in version control for, further down the road.

Sounds good. I guess eventually having SQLite support would give another option for this too as you could just put the single DB file into your set of tracked files.

[Adam Kiss]

Sounds good. I guess eventually having SQLite support would give another option for this too as you could just put the single DB file into your set of tracked files.

Actually, this could be possible right now as well - symlinking your db files into repo on one end, and symlinking your repo into your mysql dir on the other... it should work

[netcarver]

Actually, this could be possible right now as well - symlinking your db files into repo on one end, ...

Sure, but I'm pretty dumb and would probably forget to add the multiple files that MySQL creates behind the scenes for each new table I may end up adding to my database. SQLite would be easier in this respect.

As I'd probably create and modify templates more frequently than I would add tables, I probably wouldn't have the same issues remembering to add the template files to the repo.