User:Robertinventor/import process

From Encyclopedia of Buddhism
Jump to navigation Jump to search

Transferring pages and templates etc from Wikipedia

To export / import pages use:
https://en.wikipedia.org/wiki/Special:Export
set to Include templates
http://encyclopediaofbuddhism.org/index.php?title=Special:Import

Exported and imported all the templates in
http://encyclopediaofbuddhism.org/index.php?title=Special:WantedTemplates
- note that some of them are deleted in Wikipedia so it still shows as having some wanted there - and the more complete list of wanted pages also has more templates that are used by other templates and have been deleted or never existed in wikipedia

Also did the same for the ones in
http://encyclopediaofbuddhism.org/index.php?title=Special:BrokenRedirects&limit=500&offset=0

Other templates etc imported

  • For citations

Template:Cite web

  • To permit pings:

Exported / imported
Template:Ping
Template:Reply to

  • to list subpages:

Template:List subpages

  • For refs

Template:Reflist

This lead to a cite error, so fixed it by using an earlier version of the template as given here as a fix for the same error:
https://www.mediawiki.org/wiki/Topic:Tukgp77g1lwfkgfg

  • to fix an error message in Template:Pp-move-indef

Module:Pp-move-indef

  • Copyright Templates

Category:Wikipedia file copyright templates


Category:Wikipedia article copyright templates

Lua memory issue - fixed

To fix lua memory error "Lua error: Internal error: The interpreter exited with status 2" added these lines to LocalSettings.php:
$wgScribuntoEngineConf['luastandalone']['memoryLimit'] = 209715200; # bytes
$wgMaxShellMemory = 204800; # in KB
The php memory fix didn't seem to be needed
https://www.mediawiki.org/wiki/Extension:Scribunto#Lua_error:_Internal_error:_The_interpreter_exited_with_status_2

Wikimedia commons

To use Wikimedia commons,
$wgUseInstantCommons = true;
Not yet finished - shows the images as red so it's not able to access the images though it does permit local uploads of them. Probably a problem with permissions to connect to external networks in SELinux (Security Enhanced Linux):

https://stackoverflow.com/questions/12162301/instantcommons-not-working-in-mediawiki-1-19-and-selinux/30939726

They link to this issue:
http://www.lightenna.com/news/tech_report/selinux_will_mess_with_your_curl

By default 'Allow HTTPD scripts and modules to connect to the network' is unticked in 'system-config-securitylevel' ('SELinux' tab).

https://www.mediawiki.org/wiki/InstantCommons

Wikipedia attribution

For attribution, made template:
http://encyclopediaofbuddhism.org/index.php?title=En-WP-Attrib
{{En-WP-Attrib}}

Added to all the pages in:
http://encyclopediaofbuddhism.org/index.php?title=Category:Imported
For some reason that list seems incomplete so also added to:
http://encyclopediaofbuddhism.org/index.php?title=Special:AllPages

Purging

You need to purge pages here more than you do with wikipedia - after editing a template, or even adding a new page, you need to purge the page that links to it or includes it or it will show up as red linked or in case of updated template will use an older cached version.

To do that you can just edit the page and do a minor edit, e.g. add a space to it.

Or - without editing it at all, go to the history page and replace the word history in the title to purge to create a url like this:

http://encyclopediaofbuddhism.org/index.php?title=Ahimsa&action=history
change to: http://encyclopediaofbuddhism.org/index.php?title=Ahimsa&action=purge
and this will purge the cache for the page and display latest version of templates etc in it.

TO DO:

Current TO DO

  • Check SELinux is configured to let the wiki connect to the network if this is in use and if not configure it to do so.
  • Set different privileges for users and admins
  • Set up daily backups (so if there are any errors to fix then you can roll back to the most recent "good" backup in the worst case where some change in configuration seriously messes things up or if it is hacked or some such) - e.g. a new backup for each day with file name incorporating the date of the backup so that they don't overwrite. It may be possible to do this with a cron job or similar.
  • Update to the latest version of MediaWiki - may fix many of the remaining errors + means it has all the latest security patches
  • Short url - see below
  • Install the extensions from the next section if desired or any other useful extensions

That seems to be it at present. No outstanding issues to discuss until the Media Wiki upgrade when we can review the situation and see if there is anything else to fix.

Short url

This lets you change e.g.

http://encyclopediaofbuddhism.org/index.php?title=Bhavacakra

to

http://encyclopediaofbuddhism.org/wiki/Bhavacakra

so that it works like the wikipedia urls.

There are several ways to do it. My site is out of date anyway so just refer you to Mediawiki

https://www.mediawiki.org/wiki/Manual:Short_URL

Assuming you are using Apache then you can just go here and there's even a wizard to do the configuring for you

https://www.mediawiki.org/wiki/Manual:Short_URL/Apache

Extensions we can add

  • Extension:EmbedVideo is an extension that makes it very easy to embed videos from YouTube and many other providers as well as upload your own videos locally. Could be useful if we want to embed videos by Buddhist teachers on topics, for instance. I use it on my music software wikis and it works very well.
  • Extension:FlaggedRevs - this is used by the template for semiprotected status {{[[Template:{pp-pc1|{pp-pc1]]}}. We don't need this extension of course at present anyway. But it prevents imported pages from displaying an error if they have semi protected status originally.
  • Extension:DeleteBatch - I use this in my own wiki - it lets you paste a list of any number of files and click a button and it deletes them all right away.

 

tags see Boddhidharma


Tried but got errors

Tried adding. I uploaded the folders and added the necessary lines to LocalSettings.php but it couldn't find them.

This is what I added to LocalSettings.php:

wfLoadExtension( 'EmbedVideo' ); wfLoadExtension( 'DeleteBatch' ); require_once "$IP/extensions/Interwiki/Interwiki.php";wfLoadExtension( 'Interwiki' ); // To grant sysops permissions to edit interwiki data $wgGroupPermissions['sysop']['interwiki'] = true;

// To create a new user group that may edit interwiki data // (bureaucrats can add users to this group)

  1. $wgGroupPermissions['developer']['interwiki'] = true; // delete the comment indicator # as appropriate

Got errors. Trying just this on its own:

wfLoadExtension( 'EmbedVideo' );

Fatal error: Uncaught exception 'Exception' with message '/home/encyclo5/public_html/extensions/EmbedVideo/extension.json does not exist!' in /home/encyclo5/public_html/includes/registration/ExtensionRegistry.php:99 Stack trace: #0 /home/encyclo5/public_html/includes/GlobalFunctions.php(120): ExtensionRegistry->queue('/home/encyclo5/...') #1 /home/encyclo5/public_html/LocalSettings.php(214): wfLoadExtension('EmbedVideo') #2 /home/encyclo5/public_html/includes/WebStart.php(102): require_once('/home/encyclo5/...') #3 /home/encyclo5/public_html/index.php(40): require('/home/encyclo5/...') #4 {main} thrown in /home/encyclo5/public_html/includes/registration/ExtensionRegistry.php on line 99

Also tried older version:

require_once "$IP/extensions/EmbedVideo/EmbedVideo.php";

Fatal error: require_once(): Failed opening required '/home/encyclo5/public_html/extensions/EmbedVideo/EmbedVideo.php' (include_path='/home/encyclo5/public_html/vendor/pear/pear_exception:/home/encyclo5/public_html/vendor/pear/console_getopt:/home/encyclo5/public_html/vendor/pear/mail_mime-decode:/home/encyclo5/public_html/vendor/pear/net_smtp:/home/encyclo5/public_html/vendor/pear/pear-core-minimal/src:/home/encyclo5/public_html/vendor/pear/mail:/home/encyclo5/public_html/vendor/pear/mail_mime:/home/encyclo5/public_html/vendor/pear/net_socket:.:/usr/local/php56/pear') in /home/encyclo5/public_html/LocalSettings.php on line 216

Maybe there is something wrong with the path there?

But the LocalSettings.php has similar lines for wfLoadExtension( 'Cite' ); wfLoadExtension( 'CiteThisPage' ); wfLoadExtension( 'ParserFunctions' );

  1. wfLoadExtension( 'Scribunto' );

They don't trigger any error messages

Unresolved script errors

  • Script errors

There's a list of pages with errors in:
http://encyclopediaofbuddhism.org/index.php?title=Category:Pages_with_script_errors

It's worth looking through to see if there are any remaining unresolved errors. Though - sometimes a page shows an error there yet displsys fine and the errors in the templates themselves can probably be ignored if they don't cause problems in any the pages that use them. I am not sure, some of them may have script errors in wikipedia too. Some of the templates imported from Wikipedia depend on deleted templates and when I go and look in Wikipedia they are deleted there too. So - I may be missing something - but they seem to have a philosophy that if a template works no need to fix it :).

Error examples

This is a page has wiki examples for the unresolved errors below for debugging: User:Robertinventor/import process/error examples

I have discussed them on Mediawiki here: https://www.mediawiki.org/w/index.php?title=Topic:U66ixipqk6gjl91a and as of writing this then everything is in hand - Dorje108 needs to see what the hosting provider say about SELinux and upgrade to the latest MediaWiki version and then revisit any remaining errors.

Bugs to fix

  • Allow instant commons to connect to the network: - this is probably why the images from Wikimedia commons are displayed as red links rather than included as images - first thing to check anyway

Probable reason for this, need to tick:

'Allow HTTPD scripts and modules to connect to the network' in 'system-config-securitylevel' ('SELinux' tab).

Where SELinux is Security Enhanced Linux and this option defaults to unselected.

See
https://stackoverflow.com/questions/12162301/instantcommons-not-working-in-mediawiki-1-19-and-selinux/30939726
They link to this issue:
http://www.lightenna.com/news/tech_report/selinux_will_mess_with_your_curl

- it is showing the bulleted links on separate lines

Have tried to fix with 

$wgUseTidy  = true;

and with 

$wgTidyConfig = [
    'driver' => 'RaggettInternalPHP',
    'tidyConfigFile' => "$IP/includes/tidy/tidy.conf",
];

in LocalSettings.php. Neither has worked, kept the first, $wgUseTidy = true;, in place, seems worth doing anyway and this is the one that made it look most like Wikipedia html output.

  • Error in {{Good article}}: Unknown extension tag "indicator"
  • Error in Gautama Buddha semi protection template "Lua error in Module:Effective_protection_level at line 16: attempt to index field 'FlaggedRevs' (a nil value)"

Probably need to install missing extension Extension:FlaggedRevs

All it does though is to give the page Gautama Buddha semi protected status. We don't need that for now, so I commented it out from that page, for now. It is the only page here that uses it. (installation of the missing extenstion looked a bit tricky and needing care, might even introduce new bugs if done incorrectly).

When importing new pages then we need to remove the protection templates from them (or just delete the contents of the template for now if there are lots of semi protected ones to import)

  • Missing parser extension

{{Top icon}}

Error: Page status indicators' name attribute must not be empty.

Missing parser extenstion tag: <indicator>

For a listing of the ones currently installed, see Special:Version and scroll down to the extension tags section. As of writing this, this wiki currently only has five of them: <gallery>, <nowiki>, <pre>, <ref> and <references>

How many pages are there on Buddhism in Wikipedia that we could import?

I'm wondering here about what would be needed to make this reasonably complete.

Summary

Suggestion, to import the pages in {{Buddhism topics}} - about 325 needed, will slightly more than double the size of the wiki but should insure we have most of the central pages here from the Buddhism project in wikipedia.

To import all the pages would mean importing around 5000 pages which would be a long job especially with the need to add the template to each one - and also make the encyclopedia rather overwhelming to check over unless or until we have more people working on it.

Details

For an estimate of the number of pages in the Wikipedia Buddhsim project there are currently 5021 that have their talk pages tagged with the buddhism project template.

Wikipedia says there are 5,779 pages tagged as part of the Buddhism project https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Buddhism/AssessmentBox

That's not including categories - to show them all see

https://en.wikipedia.org/w/index.php?title=Special:WhatLinksHere/Template:WikiProject_Buddhism&limit=6000 (takes a while for the page to load) https://en.wikipedia.org/w/index.php?title=Special:WhatLinksHere/Template:WikiProject_Buddhism&limit=5000&from=39429234&back=0

Another way to approach this is to look at the wanted pages in this wiki:

There are over 10,100 "wanted pages" in this wiki - many could be copied over from Wikipedia - includes a fair few that don't exist in Wikipedia either. But it wouldn't take all that long to copy them all over, all the ones that exist that seem Buddhism related, a few at a time, on the other hand though that would be a lot of pages to manage and maybe we want to start smaller - and they would link to more pages too.

http://encyclopediaofbuddhism.org/index.php?title=Special:WantedPages&limit=500&offset=10000

What about the most central articles?

Well one way is to look at the Buddhism topics template {{Buddhism topics}}

Just using a line count I think it has around 394 pages it links to of which 69 are already present, so 325 to add.

So - we could export those pages from Wikipedia and import into this wiki. This wiki has about 230? pages at present (if I counted them correctly) so that would more than double the size of the wiki but it is more manageable than adding over 5,000 pages to it.

Export and import would be easy, adding the attribution template to each one would take a bit longer unless there is some way to automate it. But it is easily feasible, not a daunting task, something to finish in hours, at most, not days.

Buddhism Topics listing

I've made a list of all the pages in {{Buddhism topics}} here:

User:Robertinventor/import process/Buddhism topics

We could go through and copy over the red entries there to a list of articles to export from Wikipedia and import here. Perhaps do a few at a time.

Found some other templates we could use a similar treatement for:

Wikipedia:Template:Buddhism
User:Robertinventor/import process/Buddhism

Wikipedia:Template:Early Buddhism
User:Robertinventor/import process/Early Buddhism

Wikipedia:Template:Theravada Buddhism
User:Robertinventor/import process/Theravada Buddhism

Wikipedia:Template:Japanese Buddhism
User:Robertinventor/import process/Japanese Buddhism

_Buddhism Wikipedia:Template:Tibetan Buddhism
User:Robertinventor/import process/Tibetan Buddhism

Wikipedia:Template:Mahāyāna Buddhism
User:Robertinventor/import process/Mahāyāna Buddhism

Imported

Need attrib + check to see if we want an earlier version of any:

{{Buddhism topics}}

Attrib tags added up to here

{{En-WP-Attrib}}
{{En-WP-Attrib-date|oldid|year|monthnumber}}
{{WP-Attrib-url|//nl.wikipedia.org/wiki/Ringu_Tulku_Rinpoche|Netherlands Wikipedia|Ringu Tulku Rinpoche}}


php.ini

Currently in both /home/encyclo5 and /home/encyclo5/public_html

; Maximum size of POST data that PHP will accept.
post_max_size = 200M
upload_max_filesize =200M
max_execution_time = 4000  ; Maximum execution time of each script, in seconds
max_input_time = 4000  ; Maximum amount of time each script may spend parsing request data
; Default timeout for socket based streams (seconds)
default_socket_timeout = 4000

Was /home/encyclo5

; Maximum size of POST data that PHP will accept.
post_max_size = 20M
max_execution_time = 2000  ; Maximum execution time of each script, in seconds
max_input_time = 2000  ; Maximum amount of time each script may spend parsing request data
; Default timeout for socket based streams (seconds)
default_socket_timeout = 2000

and 0 bytes in /home/encyclo5/public_html

Robert Walker (talk) 08:52, 30 August 2018 (UTC)

Not creating thumbnails on upload images

Fixed by changing this in Localsettings.php

$wgGenerateThumbnailOnParse = true; # was false, fix 30th Aug 2018

Robert Walker (talk) 11:06, 30 August 2018 (UTC)

Using AutoWikiBrowser and updated local settings

I have been using AutoWikiBrowser - which you can use with any wiki not just Wikipedia related ones.

AutoWikiBrowser

You can use it to do automated editing, e.g. of all the pages in a category. Add or remove a category from those pages, add a template to them all or whatever. It's got a "bot" mode that auto saves each page after editing it.

The problem for a low traffic site is that - e.g. when you change a template, all the pages that transclude it have to be updated eventually. Each one is a separate job. If you add text to a hundred pages - that's 100 jobs to update those pages.

So, it adds lots of jobs to run, and they take a while to run because they run only when someone views a page on the wiki - then the next job in the queue will fun.

So I added these lines to LocalSettings.php 

 # Seems may as well run all jobs right away on the next page view,
 # for as long as there are only a couple of editors
 # easiest to do them with a high run rate
 #  https://www.mediawiki.org/wiki/Manual:$wgRunJobsAsync
 $wgRunJobsAsync = true;
 $wgJobRunRate = 1000;

This means that if we do somethng that needs lots of updates, e.g. edit a template that is linked to by many pages, then we are likely to get a 504 time out when we view a page in Wikipedia.

Immediately after doing an edit like that, just view a page and wait for it to time out. Now all those jobs are running, up to 1000.

If it causes problems just comment out those two lines. Robert Walker (talk) 17:13, 8 September 2018 (UTC)

Retrieved from "https://encyclopediaofbuddhism.org/index.php?title=User:Robertinventor/import_process&oldid=25064"