Headless ProcessWire - How Do You Do It?

[lokomotivan]

14 minutes ago, bernhard said:

I'm still curious how that "out of the box" would look like and how other platforms are working in that regard... As a detailed step by step use case.

I tired to make a workflow example using SSG and headless CMS, in previews post...

Again, what is missing is REST api "out of the box". There is no steps, you get access to all your data via end points. You create a content using proccesswire admin, u get the json feed, u update the content, json feed updates... You can take example of GraphQL module, just built in the core, and i would prefer REST. And as @flydev ?? mentioned, AppApi is a good example.

[kongondo]

17 minutes ago, lokomotivan said:

you get access to all your data via end points.

Could you please explain how this would look like? E.g you visit /mysite/admin/rest/ ? and that responds with JSON? My feeling is that we already have the tools to do this. Just trying to think of the practical implementation. 

[bernhard]

2 minutes ago, kongondo said:

Could you please explain how this would look like? E.g you visit /mysite/admin/rest/ ? and that responds with JSON? My feeling is that we already have the tools to do this. Just trying to think of the practical implementation. 

Same here. I feel like we have everything and try to understand what is missing and what others to differently (better)...

[lokomotivan]

43 minutes ago, kongondo said:

Could you please explain how this would look like? E.g you visit /mysite/admin/rest/ ? and that responds with JSON? My feeling is that we already have the tools to do this. Just trying to think of the practical implementation. 

It's not about having the tools, its about having it already build and "standardised". 

// get the single data
example.com/api/v1/page/123

// get child pages data
example.com/api/v1/pages/123

// use get varable as selector 
example.com/api/v1/pages/?template=product

// filter child pages
example.com/api/v1/pages/123?category=tv

// get user
example.com/api/v1/user/123

// get users
example.com/api/v1/users/?role=custumer

etc...

Edit: yes, this responds with json.

39 minutes ago, bernhard said:

Same here. I feel like we have everything and try to understand what is missing and what others to differently (better)...

Looking it from the back-end perspective, nothing is missing. If you try to look at the things from the front-end, it might look different.
What others are doing, is just providing external api out-of-the-box, which attracts new users especially front-end developers, and make things easier for people who are not experienced with building apis, or want to save time, so instead of processwire choosing other platforms, because they have this functionality built in.

[wbmnfktr]

I guess the most important and missing part in this thought-process might be the fact: where does the feed/API come from and how does it know what to do? Or a similar question in this direction. 

To give you more details and some new words, which get used quite often in these systems and setups. We will look at the data and CMS part here. The frontend isn't that important right now as it could vary from something like 11ty/hugo/jekyll or a full blown next/nuxt/react/whatever.js framework.

1. Content/API-first
You need to define your necessary data collections first.
These can be just posts and pages, like in WordPress, or any other kind of data you might need.
Each and every data collection is, if you will, an endpoint for your API.
Either JSON, REST, GraphQL, Markdown with Frontmatter, it depends on the system you choose.
Some have only one option, some have multiple options available.

WordPress Example:
wp.domain/wp-json/wp/v2/posts
wp.domain/wp-json/wp/v2/pages
wp.domain/wp-json/wp/v2/yourdatacollection

ProcessWire
In ProcessWire there is no such thing as data collections or static endpoints.
But we could easily create a template which indicates a data collection, like movies or restaurants, make the template single use only (Template > Family > Can this template be used for new pages?) and from there some kind of system field like Enable as API endpoint? or something like this.
With that it would be possible to have a default URL for those feeds, like:

pw.domain/api/v1/movies
pw.domain/api/v1/restaurants

If those are full featured GraphQL, REST APIs, or "just" JSON feeds at the end of the day doesn't matter right now.
As long as they all know how to handle and output image, file, repeater, matrix, combo fields.

2. How does the system know how to handle the data?
That's a thing we can already see in ProcessWire itself sometimes or in the PageQueryBoss module.
ProcessWire is aware when to output an array or just data. PageQueryBoss does the same in some cases.

These Headless CMSs I looked at have similar input field types like ProcessWire and therefore know how to show them in an API.
Text, textarea, image (single/multi), file (single/multi), Mapmarker, even repeater and matrix like fields.
The API does exactly know how to handle and what to output for those fieldtypes. Even though they are quite customizable.
You don't have to hook into anything, you don't have call another function to get an array.

It's already there and ready to use.

If it wasn't for the frontend you never have to deal with the API/Feed itself.
No loops, no functions, no hooks.

[wbmnfktr]

On 2/22/2022 at 8:37 AM, bernhard said:

1. I install Forestry on my host
2. Then I create Template XY
3. Then I add fields X, Y, Z to the template
4. Then I add Module X to get JSON Feed
5. Then I create a Frontend and access this feed

1. I choose a Headless CMS
2. I create data collections (pages, posts, movies, restaurants)
3. I add fields to match my needs in those data collections + add data/content
4. I go to headless.cms/api/v1/json|graphql|rest
5. Access it in my frontend and generate the pages

That's the short version and yes... step 4 is for that matter missing right now as core functionality.

Sure we can probably build this on our own, some already did it with a few lines of code, other with modules and even with REST API functionality, as well as with GraphQL... BUT that's always a lot of work. It's not standardized. It's a big showstopper for something you may only want to try out.

SURE... being this flexible is a huge plus as well, yet I don't buy a cow to eat a steak.

[wbmnfktr]

On 2/22/2022 at 4:07 PM, flydev ?? said:

this module is for you :  https://processwire.com/modules/blackhole/

I doubt this will work out with ProCache enabled but I'll definitely take a closer look at this. Thanks!

16 hours ago, pwired said:

Is SSG not mostly being used by coders, git-users, documentation, etc. ?

Yes and no... take a look here:
https://forestry.io/showcase/
https://www.sanity.io/projects

Even Smashing Magazine, Backlinko and Android Authority use SSG/Jamstack.

[bernhard]

Thx @wbmnfktr that helps

So what is missing with the AppApi module? I've never used it, but I like the topic and it sounds like it would be fun to build a module for it... so I'll try to understand better.

How do the other systems handle access control? How should that be done by PW? On a template level? On a field level?

1 hour ago, wbmnfktr said:

Sure we can probably build this on our own, some already did it with a few lines of code, other with modules and even with REST API functionality, as well as with GraphQL... BUT that's always a lot of work. It's not standardized. It's a big showstopper for something you may only want to try out.

I understand that. I'm a huge fan of standardizing things ? 

[wbmnfktr]

6 minutes ago, bernhard said:

So what is missing with the AppApi module?

Can't tell anything about it just because I haven't tried it so far.
And for my use cases right now I really doubt I will - even though it looks amazing.
The things necessary (installation, setup, routes, auth, ...) to get what I want is just too much.

All I really need is solid JSON for each of my data collections (pages, posts, restaurants, movies).
No need for creating or modifying content due to user input on the frontend.
No access control either.

If I wanted to build my very own all-in-one setup to manage several client projects (and way bigger ones) in this matter, I'd probably give this module a very solid try and do the things necessary. Right now for a few test runs and proof-of-concept projects I pass.

How do other systems do it?
It depends. As always.
Some have public API endpoints/ JSON feeds by default you can limit later on, similar to AppApi.
Some need request tokens you/the system define right from the start.
To limit it down they have custom roles and more. Very strict like the GraphQL module.

You might want to take a look at Custom Roles here: https://www.sanity.io/docs/access-control

Depending on your needs there are already a ton of content platforms (another word for Headless CMS for a slightly different target group) out there that provide you with various different solutions and possibilites, like author-editor-workflows and revisions.

For example: https://www.contentful.com/ or https://www.storyblok.com/

While those are pretty great for newspapers, magazines with tons of content and lots of authors those features can get pretty expensive at some point, while most others have free plans that fit a good amount of users or websites.

How should ProcessWire do it?
I guess ProcessWire already does a great job here in how it handles such things.
Reading through the AppApi docs I'd say it looks quite good how it's done there.
Still a public, non-restricted JSON feed right from the start would already get the job (or at least mine) done.

pw.com/api/v1/posts (public - defined by a global setting as described above)
pw.com/api/v1/pages (public - ...)
pw.com/api/v2/customers (with Auth to get access to restricted fields and data)
pw.com/api/v2/orders (with Auth ...)

[kongondo]

2 hours ago, wbmnfktr said:

But we could easily create a template which indicates a data collection, like movies or restaurants, make the template single use only (Template > Family > Can this template be used for new pages?) and from there some kind of system field like Enable as API endpoint? or something like this.

Quite an interesting concept you got here. I am wondering though if it would introduce more complexity? For ProcessWire, like you mentioned, posts, pages, customers, orders are all the same things. Maybe the difference would be users, i.e. $users. Other than that, all the other stuff after /api/v2/ could be query strings as per @lokomotivan's suggestion ?.

7 hours ago, lokomotivan said:

// use get varable as selector 
example.com/api/v1/pages/?template=product

 

[wbmnfktr]

19 minutes ago, kongondo said:

I am wondering though if it would introduce more complexity?

I can't answer this for all of you but looking into my projects I work with this type of concept all the time.
Just a few examples from those projects:

Home
├── ...
├─ Restaurants (template: restaurants) *
├─── Restaurant ABC (template: restaurant)
├── ...
├─ Events (template: events) *
├─── dd.mm.yyyy - Let the music play (template: event)
├── ...
├─ Teachers (template: teachers) *
├─── Mr. Mister (template: teacher)
├── ...

All pages marked with * are set to one page only and therefore could be an endpoint following that idea from above.
In terms of complexity... no, not really at all.
Should children templates be limited in this setup? Well, I do. Is it necessary not really I guess.

 

The idea from @lokomotivan would work as well but that would already be the very first point we have to take care of input. 
Sure that's still low level code but already something that's not really out of the box.
Based on that idea all endpoint-enabled templates and therefore the one and only page's name could be used here instead.
So it could either be restaurants, events, teachers (in this example) or sOm3w31rd characters we put in the page name. 

Update on this as I missed the /api/v2 part: sure, for the part of this API that needs auth that's the perfect choice I guess. You actually need it and not only one but several parameters.

[bernhard]

What about this?

https://github.com/baumrock/RockHeadless

[wbmnfktr]

That looks great!

Just gave it a try and the result looks quite similar to what I get when using your findRaw() statement.

But... YES! That's what I was thinking about. 

[bernhard]

v0.0.4 adds callbacks to easily define the returned values of your selected fields:

<?php
// in site/init.php
$rockheadless->return('rockblog_cover', function($page) {
  $images = $page->rockblog_cover;
  if(!is_array($images)) return;
  $file = $images[0]['data'];
  return "site/assets/files/{$page->id}/$file";
});

I think this should make it really simple to get a quick API while still being easy to customize.

[flydev]

19 hours ago, bernhard said:

So what is missing with the AppApi module?

Eh nothing, all is ready, we have even more with JWT implementation. It's just missing a function which return the pages tree. Something like less than 10 lines of code.

Already shipped with :

- Authentication - Three different authentication-mechanisms are ready to use.

- Access-management via UI

- Multiple different applications with unique access-rights and authentication-mechanisms can be defined

And you can generate your own routes dynamically by just adding a simple hook.

 

 

18 hours ago, wbmnfktr said:

The things necessary (installation, setup, routes, auth, ...) to get what I want is just too much.

Installation will/should be required in every case, even with a core module. But you should give a try to the AppApi module, you can start in less than two minutes, check :

 

 

[horst]

@wbmnfktr, @kongondo I just want to throw in a thought about a point I read somewhere above, regarding (native) endpoints in PW based on templates. If you think about the possible different processes only upon the access methods of an URI, reflected by $config->ajax (true|false).

You can define an output of text/html or application/json for an template. In PW, the default is text/html. Within a general hook you we can check for registered templates and then switch the output processing according to AJAX or FETCH access. Does this make any sense to you? Or have I missed the point?

 

EDIT: also upon the access method, we dynamically can switch the processing filename of the templates, not only the output type.

[kongondo]

1 hour ago, horst said:

In PW, the default is text/html. Within a general hook you we can check for registered templates and then switch the output processing according to AJAX or FETCH access. Does this make any sense to you? Or have I missed the point?

Interesting. So, with this approach we only need to register the template via a setting (on the template edit screen) and that same template can be used to output either HTML or JSON, right?

[wbmnfktr]

On 2/23/2022 at 10:38 PM, bernhard said:

I think this should make it really simple to get a quick API while still being easy to customize.

Probably... same with the other modules.
And don't get me wrong here: I don't want to build anything here. For now.
Callbacks, custom extensions, functions, fields... it's actually there in GraphQL, Pages2Json, and others.

So sure... it's possible to build it, easily, still it's yet again something that doesn't come out of the box.
I already struggle with your version right now as I have to add each and everyfield manually.
Then an entry in _init.php for all of my image fields, all of my repeater fields, all my reference fields, matrix fields...

But to be honest... in cases such like this which is somewhat a basic feature for others... I'm a bit tired at the moment.
I really enjoy ProcessWire for lots of reasons but maybe I skip it right now for this adventure.

 

 

On 2/24/2022 at 12:04 PM, flydev ?? said:

you can start in less than two minutes

Ok... wow. Reading the docs gave me the assumption it might end in a few hours to get it up and running. Wasn't there something with installing Node modules? Maybe that was another module I looked into.

I will definitely keep this in mind!

 

 

19 hours ago, horst said:

Does this make any sense to you? Or have I missed the point?

This could be absolutely handy in some cases. Your post just made me remember that optional header setting for different content types like JSON.

But as for now - even though we could already hook into some parts there - it isn't yet a solid core feature for those default feeds I mentioned earlier. Customizing is always an option in ProcessWire. But it's the same I mentioned in regards to @bernhards RockHeadless - right now I don't feel like building anything just to test new setups for the frontend.

Hence the idea to make such things a core functionality of some kind.

I will still love ProcessWire even without those feeds.

[wbmnfktr]

Yet another follow-up:

I have started to play around with more "Headless CMS" out there and... the more I use them and the more I try with them, I imagine ProcessWire not only could but would be such a perfect and way more advanced way of using "Headless CMS" with SSGs. No need to write schemas, templates, fields, conditions in a JSON, yaml, toml, or whatever file - only if you like to do so. (which we recently had a nice discussion about)

Try it on your own. Sanity, Forestry, strapi, whatever. They are nice and work really well, yet... using those I miss ProcessWire.

Not only is ProcessWire way more capable of different inputfield types and more (like Matrix, Combo), it's even more suited with lots of things in regards to access control, data sets aka "collections". Another thing is... those other CMS claim they are "Developer centric/focussed", yet their API and docs are way less easy to understand than ProcessWire. (at least from my non-developer perspective)

For example and without naming names:
Imagine a code block that's a default for a module or default setting, but it doesn't work because it's from 2019 and doesn't work anymore since at least 4 main/stable releases of that very "Headless CMS". To be fair... we maybe could find such things here as well, yet... often most of their forums/communities and support are located somewhere on Slack, Discord, StackOverflow, Github or are overall very thin - to say it mildly. (Ignore those stars on Github)

Which brings me straight back to my original question: How to export those data in a comfortable way through JSON, Rest API, GraphQL.

[wbmnfktr]

In case you are interested in my thoughts... I started to write down a kind of a concept about this - not yet complete by any means!

Google Drive: https://docs.google.com/document/d/e/2PACX-1vSTT9seNtgtB2AzKaMx_xZl9S1HvsR00wrlfPWhtu6D-_sfYacCO0Cbs_PT6KnUTXh8F7EahjSTgEXn/pub

You can't edit the document, but I might move this to Github/Markdown where you can fork and therefore add pull requests... but for now this is my collection of thoughts.

[bernhard]

I still don't get what you are looking for. I thought I understand your need: Making the process more standardized and easier for non-devs meaning more click click instead of writing code. That's what I tried to solve with my proof of concept module. But you said "that looks great" in your first post and then turned around and it does not seem to be what you are wanting...

On 2/24/2022 at 12:04 PM, flydev ?? said:

On 2/23/2022 at 5:56 PM, bernhard said:

So what is missing with the AppApi module?

Eh nothing, all is ready, we have even more with JWT implementation. It's just missing a function which return the pages tree. Something like less than 10 lines of code.

This question was targeted to @wbmnfktr to better understand him. When we where talking about easy json feed the very first time my answer was: Url hook + findRaw + json_encode... ( https://processwire.com/talk/topic/19112-module-page-query-boss/?do=findComment&comment=221874 )

On 2/24/2022 at 12:04 PM, flydev ?? said:

But you should give a try to the AppApi module, you can start in less than two minutes, check :

I don't agree on this one. I have not tried the module (because I have not had the need), but it looks complicated to me. Something that definitely would take longer than 2 min even just to understand how to set it up. That's why I built my module - super simple to setup, no code necessary.

But it is still not what @wbmnfktr is looking for, so I'm confused and it feels like we are going in circles...

[flydev]

On my side, from what I understand (and what I would like to see implemented) is something like the following :

- in config.php set $config->restapi = [ 'endpoint' => 'api', 'enabled' => true];

then use simples built-in functions to get data from api (eg, home page) GET /api/1

Anyway, even if it could/should be easy to use, things are a bit more complex than what we can see here in this thread. I mean, almost all functionalities of AppApi should definitely exists.

 

 

 

 

[gmclelland]

I could be wrong, but what I think @wbmnfktr is looking for is a **standardized** Processwire APIs across all Processwire installations that is on by default?

This could be beneficial by allowing:

• third party services to integrate easily with Processwire.  Something like zapier.com could build a Processwire connector that consumes the API to allow for no-code workflows that connect different systems and services together?
• a site aggregator website that could consume the other Processwire website's API and report back the details.  For example, which sites need module or Processwire updates.  Something like https://sitedash.app/ for Modx
• Static Site Generators to consume and build a fast static website that can be hosted on a global CDN.
• a Single Page Application built with Vue.js/React.js/React Native, etc.. that could be replace the Processwire Admin.  I think https://www.sanity.io/ can do this?  Everything is fully decoupled.  Why would you want a different admin?  What if you wanted to build a native Mobile app to administer your Processwire site?
• admin components that consume that consume the API for different admin experiences?  Wordpress uses the API for their new Block Editor https://developer.wordpress.org/block-editor/

Sure stuff like sitedash.app can be built right now with Processwire, but services like zapier.com and others aren't going to spend time building a API connector if it isn't included in Processwire core and isn't standardized.

I agree with flydev - there are other things to consider as well like issuing API tokens, content throttling, API versioning, providing data in different formats other than json and REST like GraphQL, webhooks, autogenerated API documentation like https://swagger.io/.  https://api-platform.com/ covers a lot of these topics.

https://strapi.io/ does a good job with some of these things like issuing tokens for integrating third party clients.

Thanks everyone for posting solutions that could work.  I enjoying reading and watching the many different ways you can do things with and without modules.  Thanks @flydev ?? for the AppAPI demo.  Thanks @bernhard for showing/creating the RockHeadless module and demo - dang your fast.  I like how you demonstrate how you can also expose the children of certain pages to the API as well.  That's is another aspect that has to be considered since Processwire is different than most bucket based CMSs.  Processwire is tree based around hierarchy.

 

[Sebi]

Hello all!

I guess I'm a little late to the party, but here's an explanation to my AppApi approach. I basically wanted to achieve exactly the same thing with my projects as well: A universal JSON api that I make all public ProcessWire pages also queryable as JSON. 

For this I use the combination of my Twack module and AppApi. Twack is a component system that allows me to get a JSON output from each ProcessWire page in addition to the standard HTML output. Twack registers a route that allows to query each Page from the PageTree (via ID or Path).

But for this route you don't need the Twack module, of course. As here it would work only with the AppApi module:

routes.php:

<?php

namespace ProcessWire;

require_once wire('config')->paths->AppApi . 'vendor/autoload.php';

$routes = [
	'page' => [
		['OPTIONS', '{id:\d+}', ['GET', 'POST', 'UPDATE', 'DELETE']],
		['OPTIONS', '{path:.+}', ['GET', 'POST', 'UPDATE', 'DELETE']],
		['OPTIONS', '', ['GET', 'POST', 'UPDATE', 'DELETE']],
		['GET', '{id:\d+}', PageApiAccess::class, 'pageIDRequest'],
		['GET', '{path:.+}', PageApiAccess::class, 'pagePathRequest'],
		['GET', '', PageApiAccess::class, 'dashboardRequest'],
		['POST', '{id:\d+}', PageApiAccess::class, 'pageIDRequest'],
		['POST', '{path:.+}', PageApiAccess::class, 'pagePathRequest'],
		['POST', '', PageApiAccess::class, 'dashboardRequest'],
		['UPDATE', '{id:\d+}', PageApiAccess::class, 'pageIDRequest'],
		['UPDATE', '{path:.+}', PageApiAccess::class, 'pagePathRequest'],
		['UPDATE', '', PageApiAccess::class, 'dashboardRequest'],
		['DELETE', '{id:\d+}', PageApiAccess::class, 'pageIDRequest'],
		['DELETE', '{path:.+}', PageApiAccess::class, 'pagePathRequest'],
		['DELETE', '', PageApiAccess::class, 'dashboardRequest']
	]
];

You can use this class to get the page-outputs:

<?php

namespace ProcessWire;

class PageApiAccess {
	public static function pageIDRequest($data) {
		$data = AppApiHelper::checkAndSanitizeRequiredParameters($data, ['id|int']);
		$page = wire('pages')->get('id=' . $data->id);
		return self::pageRequest($page);
	}

	public static function dashboardRequest() {
		$page = wire('pages')->get('/');
		return self::pageRequest($page);
	}

	public static function pagePathRequest($data) {
		$data = AppApiHelper::checkAndSanitizeRequiredParameters($data, ['path|pagePathName']);
		$page = wire('pages')->get('/' . $data->path);
		return self::pageRequest($page);
	}

	protected static function pageRequest(Page $page) {
		$lang = SELF::getLanguageCode(wire('input')->get->pageName('lang'));
		if (!empty($lang) && wire('languages')->get($lang)) {
			wire('user')->language = wire('languages')->get($lang);
		} else {
			wire('user')->language = wire('languages')->getDefault();
		}

		if (!$page->viewable()) {
			throw new ForbiddenException();
		}

		return $page->render();
	}

	private static function getLanguageCode($key) {
		$languageCodes = [
			'de' => 'german',
			'en' => 'english'
		];

		$code = '' . strtolower($key);
		if (!empty($languageCodes[$key])) {
			$code = $languageCodes[$key];
		}

		return $code;
	}
}

On top of your ProcessWire-template file you have to check if an api-call is active. If so, output JSON instead of HTML:

<?php
// Check if AppApi is available:
if (wire('modules')->isInstalled('AppApi')) {
  $module = $this->wire('modules')->get('AppApi');
  // Check if page was called via AppApi
  if($module->isApiCall()){
    // Output id & name of current page
    $output = [
      'id' => wire('page')->id,
      'name' => wire('page')->name
    ];
    
    // sendResponse will automatically convert $output to a JSON-string:
    AppApi::sendResponse(200, $output);
  }
}

// Here continue with your HTML-output logic...

That should be everything necessary to enable your ProcessWire-templates to output JSON. You will have an /api/page/ endpoint, which can be called to get the JSON-outputs.

/api/page/test/my-page -> ProcessWire page 'my-page' in your page-tree under test/ 
/api/page/42  -> ProcessWire page with id 42
/api/page/ -> root-page

And a small addition:
If you want to query the page files also via api, have a look at my AppApiFile module. With it you can query all images via an /api/file/ interface.

Edited March 6, 2022 by Sebi
Missing AppApi::sendResponse() param

[Sebi]

I just released a new extension module AppApiPage (waits for approval), which handles the initial steps from my post above completely automatic.

You install AppApi and AppApiPage. That makes the /api/page route available and you only have to add the code on top of your template php to add a custom JSON output to your pages.

<?php
// Check if AppApi is available:
if (wire('modules')->isInstalled('AppApi')) {
  $module = $this->wire('modules')->get('AppApi');
  // Check if page was called via AppApi
  if($module->isApiCall()){
    // Output id & name of current page
    $output = [
      'id' => wire('page')->id,
      'name' => wire('page')->name
    ];
    
    // sendResponse will automatically convert $output to a JSON-string:
    AppApi::sendResponse(200, $output);
  }
}

// Here continue with your HTML-output logic...

I hope that this makes it even simpler to add a full-blown JSON api to new and existing pages.