Modules Directory code for parsing module info

[Robin S]

This might be a question only @ryan can answer, or maybe someone know if Ryan has ever shared the code used in the PW Modules Directory...

I want to make a module similar to ProcessWireUpgrade except for private modules that are not in the modules directory. I haven't decided yet exactly how this will work (get each module info on demand direct from the repo or set up a private modules directory that gets several module's info on a lazycron), but part of it will need to be parsing the version number from the getModuleInfo() code in the module repo.

I can probably muddle through this but rather than reinvent the wheel, is it possible to see how the PW Modules Directory does it?

[adrian]

Not sure but I think @netcarver has been doing some work along the lines of finding unpublished modules, so maybe there is some crossover or code that can be shared?

[netcarver]

I do have a script that trawls github and packagist for projects that are not in the PW repo. It's poor quality, but it does the job. In the screenshot below, it's found 1,224 projects - only 557 of which are in the PW Repo.  Filtering of results is pretty nice, too.

By adding some timestamps (when project is first seen and then last modified) this could be used as the basis for detecting changes to unregistered modules and projects. 

[Robin S]

1 hour ago, netcarver said:

I do have a script that trawls github and packagist for projects that are not in the PW repo.

Looks cool!

Does your script parse out the version number from the getModuleInfo() method / ModuleName.info.php file / ModuleName.info.json file? If so, would you be willing to share the code from that part? I have an idea in mind for how to do this but it would be no surprise if your idea was better. ?

[netcarver]

Hi Robin,

I originally envisioned PW Gems as casting a wider net than it currently does, with search methods for additional PW sources including Google, github gists, code snippets embedded here in forum posts, as well as public gitlab and bitbucket repos. I only got around to doing the PW repo, github and packagist searches in the end and, of these, only the PW Repo search extracts version information.

There's no reason these searches couldn't dive deeper and look for versions from well known files, but I've not done it. Of course, not all the projects turned up are modules. Many of the repos are people's PW site boilerplates for example, though they may have a published tag or release that contains a version number.

[Robin S]

For those interested, here is a proof-of-concept Process module for parsing a module version number from a GitLab repo, public or private (if you have an access token). For demonstration there is a form for entering the module info and displaying the retrieved version, but for real usage you would send the module info to the Process page by POST and get the version number back in an AJAX response.
 

Spoiler

<?php namespace ProcessWire;

class ProcessGitlabVersion extends Process {

	public static function getModuleinfo() {
		return array(
			'title' => 'Gitlab Version',
			'summary' => 'Gets the version number of a ProcessWire module from a Gitlab repository.',
			'version' => '0.1.0',
			'author' => 'Robin Sallis',
			'icon' => 'info-circle',
			'page' => array(
				'name' => 'gitlab-version',
				'title' => 'Gitlab Version',
				'parent' => 'setup'
			),
		);
	}

	public function ___execute() {

		// Get relevant data from input
		$input = $this->wire()->input;
		$module_name = $input->post->text('module_name');
		$repo_url = $input->post->url('repo_url');
		$token = $input->post->text('token');

		// Form fields
		/* @var InputfieldForm $f */
		$form = $this->wire()->modules->InputfieldForm;

		/* @var InputfieldText $f */
		$f = $this->wire()->modules->InputfieldText;
		$f->name = 'module_name';
		$f->label = $this->_("Module name");
		$f->value = $module_name;
		$form->add($f);

		/* @var InputfieldText $f */
		$f = $this->wire()->modules->InputfieldText;
		$f->name = 'repo_url';
		$f->label = $this->_("URL to module's GitLab repo");
		$f->value = $repo_url;
		$form->add($f);

		/* @var InputfieldText $f */
		$f = $this->wire()->modules->InputfieldText;
		$f->name = 'token';
		$f->label = $this->_("GitLab access token");
		$f->value = $token;
		$form->add($f);


		// Process any submitted form data
		if($module_name && $repo_url) {
			// Prepare project ID for GitLab API
			$project_id = str_replace('https://gitlab.com/', '', $repo_url);
			$project_id = urlencode($project_id);

			$http = new WireHttp();
			$http->setValidateURLOptions(['convertEncoded' => false]);

			$version_file = $this->getVersionFile($http, $module_name, $project_id, $token);
			// If a suitable file was found
			if(!empty($version_file)) {
				$version_string = $this->getVersionString($http, $project_id, $version_file);
				// If a version string was found
				if($version_string) {
					/* @var InputfieldMarkup $f */
					$f = $this->wire()->modules->InputfieldMarkup;
					$f->label = $this->_("Version number");
					$f->value = $version_string;
					$form->add($f);
				}
			}
		}

		// Add submit button and render the form
		/* @var InputfieldSubmit $f */
		$f = $this->wire()->modules->InputfieldSubmit;
		$form->add($f);
		return $form->render();

	}

	/**
	 * Get file from repo that contains the version string
	 *
	 * @param WireHttp $http
	 * @param string $module_name
	 * @param string $project_id
	 * @param string $token
	 * @return array
	 */
	protected function getVersionFile($http, $module_name, $project_id, $token = '') {

		if($token) $http->setHeader('PRIVATE-TOKEN', $token);

		// Get repo tree
		$tree_url = "https://gitlab.com/api/v4/projects/$project_id/repository/tree";
		$data = $http->getJSON($tree_url);

		// Find the file containing the version string
		$version_file = array();
		$module_name_lower = strtolower($module_name);
		foreach($data as $file) {
			if($file['type'] !== 'blob') continue;
			$file_name = strtolower($file['name']);
			if($file_name === "$module_name_lower.info.php") {
				$version_file['name'] = $file['name'];
				$version_file['type'] = 'info_php';
				break;
			}
			if($file_name === "$module_name_lower.info.json") {
				$version_file['name'] = $file['name'];
				$version_file['type'] = 'info_json';
				break;
			}
			if($file_name === "$module_name_lower.module" || $file_name === "$module_name_lower.module.php") {
				$version_file['name'] = $file['name'];
				$version_file['type'] = 'module';
			}
		}

		return $version_file;
	}

	/**
	 * Get version string from file contents
	 *
	 * @param WireHttp $http
	 * @param string $project_id
	 * @param array $version_file
	 * @return string
	 */
	protected function getVersionString($http, $project_id, $version_file) {
		$version_string = '';

		// Get file content from GitLab
		$file_url = "https://gitlab.com/api/v4/projects/$project_id/repository/files/{$version_file['name']}?ref=master";
		$data = $http->getJSON($file_url);
		$content = base64_decode($data['content']);

		// For info.json file, get version string directly by decoding JSON
		if($version_file['type'] === 'info_json') {
			$content = wireDecodeJson($content);
			if(isset($content['version'])) return $content['version'];
		}

		// Will the next occurrence of a matching version line be used?
		$next_occurrence = $version_file['type'] === 'info_php';
		
		// Loop over lines in file content
		$separator = "\r\n";
		$line = strtok($content, $separator);
		while($line !== false) {
			// For module file, use the next occurrence of a version line after getModuleinfo()
			if($version_file['type'] === 'module' && strpos($line, 'getModuleInfo()') !== false) {
				$next_occurrence = true;
			}
			if($next_occurrence && $version_string = $this->getVersionFromLine($line)) break;
			$line = strtok($separator);
		}

		return $version_string;
	}

	/**
	 * Get the version number from a line of code
	 *
	 * @param string $str
	 * @return bool
	 */
	protected function getVersionFromLine($str) {
		// Return early if "version" is not present
		if(strpos($str, '"version"') === false && strpos($str, "'version'") === false) return '';
		$pieces = explode('=>', $str, 2);
		// Return early if "=>" is not present
		if(count($pieces) < 2) return '';
		$version_string = strtok($pieces[1], '/'); // discard any slashed comments
		$version_string = strtok($version_string, '#'); // discard any hashed comments
		$version_string = trim($version_string, "\"', "); // trim quotes, commas, spaces
		return $version_string;
	}

}

 

 

Form...

...and with version number returned...

[netcarver]

Nice, thank you for working on the GitLab side of things. Would you mind if I used your code as the basis of a GitLab adaptor for the Release Notes module?

PS> There is already code in ReleaseNotes for both Github and BitBucket, so adding the GitLab adaptor would give some nice closure to ReleaseNotes. 

[Robin S]

An updated version supporting both GitHub and GitLab:
(not sure if the GitHub authentication works because I don't have the paid account needed in order to create a private repo)

Spoiler

<?php namespace ProcessWire;

class ProcessRepoVersion extends Process {

	public static function getModuleinfo() {
		return array(
			'title' => 'Repo Version',
			'summary' => 'Gets the version number of a ProcessWire module from a GitHub/GitLab repository.',
			'version' => '0.1.0',
			'author' => 'Robin Sallis',
			'icon' => 'info-circle',
			'page' => array(
				'name' => 'repo-version',
				'title' => 'Repo Version',
				'parent' => 'setup'
			),
		);
	}

	public function ___execute() {

		// Get relevant data from input
		$input = $this->wire()->input;
		$module_name = $input->post->text('module_name');
		$repo_url = $input->post->url('repo_url');
		$token = $input->post->text('token');

		// Form fields
		/* @var InputfieldForm $f */
		$form = $this->wire()->modules->InputfieldForm;

		/* @var InputfieldText $f */
		$f = $this->wire()->modules->InputfieldText;
		$f->name = 'module_name';
		$f->label = $this->_("Module class name");
		$f->value = $module_name;
		$f->required = 1;
		$form->add($f);

		/* @var InputfieldText $f */
		$f = $this->wire()->modules->InputfieldText;
		$f->name = 'repo_url';
		$f->label = $this->_("URL to module's GitHub/GitLab repository");
		$f->value = $repo_url;
		$f->required = 1;
		$form->add($f);

		/* @var InputfieldText $f */
		$f = $this->wire()->modules->InputfieldText;
		$f->name = 'token';
		$f->label = $this->_("Access token (if needed)");
		$f->value = $token;
		$form->add($f);
		
		// Process any submitted form data
		if($module_name && $repo_url) {

			if(strpos($repo_url, 'github')) {
				$version_string = $this->versionFromGitHub($module_name, $repo_url, $token);
			} elseif(strpos($repo_url, 'gitlab')) {
				$version_string = $this->versionFromGitLab($module_name, $repo_url, $token);
			} else {
				$version_string = '';
				$this->error($this->_("Invalid respository. Only GitHub and GitLab repositories are supported."));
			}

			// Add markup field if a version string was found
			if($version_string) {
				/* @var InputfieldMarkup $f */
				$f = $this->wire()->modules->InputfieldMarkup;
				$f->label = $this->_("Version number");
				$f->value = $version_string;
				$form->add($f);
			} else {
				$this->warning($this->_("Failed to acquire version number."));
			}
		}

		// Add submit button and render the form
		/* @var InputfieldSubmit $f */
		$f = $this->wire()->modules->InputfieldSubmit;
		$form->add($f);
		return $form->render();

	}

	/**
	 * Get version string from GitHub module
	 *
	 * @param string $module_name
	 * @param string $repo_url
	 * @param string $token
	 * @return string
	 */
	protected function versionFromGitHub($module_name, $repo_url, $token = '') {
		$http = new WireHttp();

		// Set token if present (not sure this works and have no way to test it)
		if($token) $http->setHeader('Authorization', 'token ' . $token);

		// Project ID
		$project_id = str_replace('https://github.com/', '', $repo_url);

		// Get contents of repo
		$files = $http->getJSON("https://api.github.com/repos/$project_id/contents");
		if(empty($files)) {
			$this->warning($this->_("Failed to list repository files."));
			return '';
		}

		// Identify version file
		$version_file = $this->identifyVersionFile($files, $module_name);

		// Get file data
		$file_data = $http->getJSON("https://api.github.com/repos/$project_id/contents/{$version_file['name']}");
		if(empty($file_data)) {
			$this->warning($this->_("Failed to retrieve file data."));
			return '';
		}
		$file_contents = base64_decode($file_data['content']);

		// Return version string
		return $this->getVersionString($file_contents, $version_file);
	}

	/**
	 * Get version string from GitLab module
	 *
	 * @param string $module_name
	 * @param string $repo_url
	 * @param string $token
	 * @return string
	 */
	protected function versionFromGitLab($module_name, $repo_url, $token = '') {
		$http = new WireHttp();

		// Set token if present
		if($token) $http->setHeader('PRIVATE-TOKEN', $token);

		// Project ID
		$project_id = str_replace('https://gitlab.com/', '', $repo_url);
		$project_id = urlencode($project_id);

		// Need to keep URL encoding for GitLab project ID
		$http->setValidateURLOptions(['convertEncoded' => false]);

		// Get contents of repo
		$files = $http->getJSON("https://gitlab.com/api/v4/projects/$project_id/repository/tree");
		if(empty($files)) {
			$this->warning($this->_("Failed to list repository files."));
			return '';
		}

		// Identify version file
		$version_file = $this->identifyVersionFile($files, $module_name);

		// Get file data
		$file_data = $http->getJSON("https://gitlab.com/api/v4/projects/$project_id/repository/files/{$version_file['name']}?ref=master");
		if(empty($file_data)) {
			$this->warning($this->_("Failed to retrieve file data."));
			return '';
		}
		$file_contents = base64_decode($file_data['content']);

		// Return version string
		return $this->getVersionString($file_contents, $version_file);
	}

	/**
	 * Identify file that will contain the version string
	 *
	 * @param array $files
	 * @param string $module_name
	 * @return array
	 */
	protected function identifyVersionFile($files, $module_name) {
		$version_file = array();
		$module_name_lower = strtolower($module_name);
		foreach($files as $file) {
			if($file['type'] !== 'blob' && $file['type'] !== 'file') continue;
			$file_name = strtolower($file['name']);
			if($file_name === "$module_name_lower.info.php") {
				$version_file['name'] = $file['name'];
				$version_file['type'] = 'info_php';
				break;
			}
			if($file_name === "$module_name_lower.info.json") {
				$version_file['name'] = $file['name'];
				$version_file['type'] = 'info_json';
				break;
			}
			if($file_name === "$module_name_lower.module" || $file_name === "$module_name_lower.module.php") {
				$version_file['name'] = $file['name'];
				$version_file['type'] = 'module';
			}
		}
		return $version_file;
	}

	/**
	 * Get version string from file contents
	 *
	 * @param string $file_contents
	 * @param array $version_file
	 * @return string
	 */
	protected function getVersionString($file_contents, $version_file) {
		$version_string = '';

		// For info.json file, get version string directly by decoding JSON
		if($version_file['type'] === 'info_json') {
			$info_array = wireDecodeJson($file_contents);
			if(isset($info_array['version'])) return $info_array['version'];
		}

		// Will the next occurrence of a matching version line be used?
		$next_occurrence = $version_file['type'] === 'info_php';
		
		// Loop over lines in file content
		$separator = "\r\n";
		$line = strtok($file_contents, $separator);
		while($line !== false) {
			// For module file, use the next occurrence of a version line after getModuleinfo()
			if($version_file['type'] === 'module' && strpos($line, 'getModuleInfo()') !== false) {
				$next_occurrence = true;
			}
			if($next_occurrence && $version_string = $this->getVersionFromLine($line)) break;
			$line = strtok($separator);
		}

		return $version_string;
	}

	/**
	 * Get the version number from a line of code
	 *
	 * @param string $str
	 * @return bool
	 */
	protected function getVersionFromLine($str) {
		// Return early if "version" is not present
		if(strpos($str, '"version"') === false && strpos($str, "'version'") === false) return '';
		$pieces = explode('=>', $str, 2);
		// Return early if "=>" is not present
		if(count($pieces) < 2) return '';
		$version_string = strtok($pieces[1], '/'); // discard any slashed comments
		$version_string = strtok($version_string, '#'); // discard any hashed comments
		$version_string = trim($version_string, "\"', "); // trim quotes, commas, spaces
		return $version_string;
	}

}

 

 

3 hours ago, netcarver said:

Would you mind if I used your code as the basis of a GitLab adaptor for the Release Notes module?

Sure, that's fine.