Infrastructure

From OpenSimulator

(Difference between revisions)
Jump to: navigation, search
(Todo)
 
(11 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
__NOTOC__
 
__NOTOC__
{{Template:Quicklinks}}
+
{{Quicklinks}}
 
<br />
 
<br />
  
This page is designed to keep track of the current infrastructure running the opensim project. If you have questions or suggestions, please use the Talk page.
+
This page is designed to keep track of the current infrastructure running the OpenSim project. If you have questions or suggestions, please use the Talk page.
  
 
= Current Infrastructure =
 
= Current Infrastructure =
  
 
== Wiki ==
 
== Wiki ==
The wiki is currently Mediawiki 1.13.3. Mediawiki should be kept up to date, look for security updates once a month or so on this.
+
The wiki is currently Mediawiki 1.19.9. Mediawiki should be kept up to date, look for security updates once a month or so on this.
  
 
=== Search for Wiki ===
 
=== Search for Wiki ===
  
Search for the wiki is provided by MWSearch, which requires a seperate lucene process running. Lucene is a java program that generates a full text search engine for a set of data. It does this for media wiki by doing db dumps on a periodic basis. MWSearch is flakey, and some times needs to be whacked and restarted.
+
Search for the wiki is provided by MWSearch, which requires a separate Lucene process running. Lucene is a java program that generates a full-text search engine for a set of data. It does this for media wiki by doing DB dumps on a periodic basis. MWSearch is flakey and sometimes needs to be whacked and restarted.  
  
MWSearch lives in /var/www/lucene-search-2.1. You currently need to run ./build on a regular basis there because OAI extension is not yet correctly configured to do incremental updates.
+
MWSearch lives in /var/www/lucene-search-2.1. You currently need to run ./build on a regular basis there because OAI extension is not yet correctly configured to do incremental updates.
  
 
== Mantis ==
 
== Mantis ==
We are currently running a patched mantis 1.1.1. It has patches added for additional fields for patch flow (the green states), and to write to a named pipe in /var/run/mantis/mantis-notify with bug change information. This named pipe is read by the osmantis bot. Mantis has not been upgraded in a while because 1.2.x looks like it's on the near horizon, and it would be better to update to that if possible.
+
[http://opensimulator.org/mantis OpenSimulator Mantis Bug Tracker]
 +
 
 +
We are currently running a patched mantis 1.1.1. It has patches added for additional fields for patch flow (the green states), and to write to a named pipe in /var/run/mantis/mantis-notify with bug change information. This named pipe is read by the mantis bot. Mantis has not been upgraded in a while because 1.2.x looks like it's on the near horizon, and it would be better to update to that if possible.
  
 
=== osmantis bot ===
 
=== osmantis bot ===
  
This is a piece of perl code hacked together quickly by sean dague based on a an example perl irc bot using the POE framework. It monitors the panda rss feed for build results, and the named pipe for mantis changes, reformats the messages, and sticks them back out onto IRC. While by no means elegant, it does get the job done. The source code for this is at http://github.com/sdague/osmantis/tree/master. It must be run inside a screen session, and does not currently reconnect in the event of loosing connection with the irc server.
+
This is a piece of Perl code hacked together quickly by Sean Dague based on an example Perl IRC bot using the POE framework. It monitors the panda RSS feed for build results, and the named pipe for mantis changes reformats the messages and sticks them back out onto IRC. While by no means elegant, it does get the job done. The source code for this is at http://github.com/sdague/osmantis/tree/master. It must be run inside a screen session and does not currently reconnect in the event of losing connection with the IRC server.
  
 
== Mailing Lists ==
 
== Mailing Lists ==
These are on berlios.de because the current opensimulator.org box sits on an ip that is in a lot of spam black lists.  For this reason many people don't get mantis emails. Networks that are especially bad about that are yahoo.com and pacbell.net
+
* [http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users opensim-users] - For OpenSimulator users.
 +
* [http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev opensim-dev] - For OpenSimulator developers.
  
 
== Source Management ==
 
== Source Management ==
 +
 +
[http://opensimulator.org/viewgit/ Browse OpenSimulator Git repos]
 +
 +
See [[Source_Code_Repository|Git Source Code Repository]] for details on cloning
  
 
=== git ===
 
=== git ===
  
The git repository for opensim is in /var/git/opensim. This has been initialized with --shared=group which keeps group writable permissions on it. The opensim-svn group is the shared writable group for that repo.
+
The git repository for OpenSim is in /var/git/OpenSim. This has been initialized with --shared=group which keeps group writable permissions on it. The OpenSim-svn group is the shared writable group for that repo.
  
git:// urls are served via the git-daemon package on debian, which is run as part of the init process.
+
git:// URLs are served via the git-daemon package on Debian, which is run as part of the init process.
  
 
=== git emails ===
 
=== git emails ===
  
Commit emails are sent out via the post-receive hook (.git/hooks/post-receive) using the post-receive-email script. Some jiggling of references is required to support both email and ciabot.
+
Commit emails are sent out via the post-receive hook (.git/hooks/post-receive) using the post-receive-email script. Some jiggling of references is required to support both email and ciabot.
  
 
=== git to cia bot ===
 
=== git to cia bot ===
  
Using the post-receive hook every commit is pushed to the ciabot using .git/hooks/ciabot.pl. This uses the xml-rpc interface for cia.
+
Using the post-receive hook every commit is pushed to the ciabot using .git/hooks/ciabot.pl. This uses the xml-rpc interface for cia.
  
 
=== mirroring to github ===
 
=== mirroring to github ===
  
The opensim-mirror id is used to mirror the opensim git repo out to github every 10 minutes.  This is one directional, and done under cron.
+
[https://github.com/opensim/opensim OpenSimulator Github mirror]
  
=== panda ===
+
The opensim-mirror id is used to mirror the OpenSim git repo out to GitHub every 10 minutes. This is one directional and done under cron.
  
Panda is configured in /var/www/panda.opensimulator.org. Panda was written from scratch by sean dague to meet the unit testing needs of opensim which none of the other tools were meeting. It is separately maintained at http://projects.dague.net.  Most panda configuration is done via the database, though that isn't nicely exposed administratively yet.
+
=== Unit Testing ===
 +
 
 +
[http://jenkins.opensimulator.org/ Jenkin OpenSimulator Dashboard]
 +
 
 +
Jenkins test results are announced on the #opensim-dev IRC channel
 +
 
 +
Orginally, test builds were done on Bamboo. Then Bamoboo was replaced by Panda, a custom unit test tool developed by Sean Dague specifically for OpenSimulator. Today, OpenSimulator has retired Panda and is using Jenkins for test builds. More details can be found on the [[Jenkins]] page.
  
 
=== viewgit ===
 
=== viewgit ===
Line 56: Line 69:
 
=== svn tracking mirror ===
 
=== svn tracking mirror ===
  
The opensim-mirror id also uses a very crude script to attempt to create an svn mirror of the git repo. This is quite error prone, as svn seems to love to remove permissions on executables. Hopefully this will eventually be retired.
+
The OpenSim-mirror id also uses a very crude script to attempt to create an svn mirror of the git repo. This is quite an error-prone program, as svn seems to love to remove permissions on executables. Hopefully, this will eventually be retired.
  
 
== Backups ==
 
== Backups ==
Line 68: Line 81:
 
** /var/dbbackups
 
** /var/dbbackups
  
Backuppc has aged backups, so we've got dailies for a week, weeklies for a month, monthlies for 4 months, and every 4 months for 16 months. After the data losses on the previous box, good backups were a must.
+
Backuppc has aged backups, so we've got dailies for a week, weeklies for a month, monthlies for 4 months, and every 4 months for 16 months. After the data losses on the previous box, good backups were a must.
  
 
= Todo =
 
= Todo =
  
* Mediawiki Upgrade
+
* Mediawiki Upgrade (and switch to https protocol?)
* Find new backup solution
+
* Find a new backup solution
 
+
* Publish osmantis bot to opensim forge
+
 
* Upgrade mantis to latest stable
 
* Upgrade mantis to latest stable
 
[[Category:Infrastructure]]
 

Latest revision as of 10:03, 20 October 2021


This page is designed to keep track of the current infrastructure running the OpenSim project. If you have questions or suggestions, please use the Talk page.

[edit] Current Infrastructure

[edit] Wiki

The wiki is currently Mediawiki 1.19.9. Mediawiki should be kept up to date, look for security updates once a month or so on this.

[edit] Search for Wiki

Search for the wiki is provided by MWSearch, which requires a separate Lucene process running. Lucene is a java program that generates a full-text search engine for a set of data. It does this for media wiki by doing DB dumps on a periodic basis. MWSearch is flakey and sometimes needs to be whacked and restarted.

MWSearch lives in /var/www/lucene-search-2.1. You currently need to run ./build on a regular basis there because OAI extension is not yet correctly configured to do incremental updates.

[edit] Mantis

OpenSimulator Mantis Bug Tracker

We are currently running a patched mantis 1.1.1. It has patches added for additional fields for patch flow (the green states), and to write to a named pipe in /var/run/mantis/mantis-notify with bug change information. This named pipe is read by the mantis bot. Mantis has not been upgraded in a while because 1.2.x looks like it's on the near horizon, and it would be better to update to that if possible.

[edit] osmantis bot

This is a piece of Perl code hacked together quickly by Sean Dague based on an example Perl IRC bot using the POE framework. It monitors the panda RSS feed for build results, and the named pipe for mantis changes reformats the messages and sticks them back out onto IRC. While by no means elegant, it does get the job done. The source code for this is at http://github.com/sdague/osmantis/tree/master. It must be run inside a screen session and does not currently reconnect in the event of losing connection with the IRC server.

[edit] Mailing Lists

[edit] Source Management

Browse OpenSimulator Git repos

See Git Source Code Repository for details on cloning

[edit] git

The git repository for OpenSim is in /var/git/OpenSim. This has been initialized with --shared=group which keeps group writable permissions on it. The OpenSim-svn group is the shared writable group for that repo.

git:// URLs are served via the git-daemon package on Debian, which is run as part of the init process.

[edit] git emails

Commit emails are sent out via the post-receive hook (.git/hooks/post-receive) using the post-receive-email script. Some jiggling of references is required to support both email and ciabot.

[edit] git to cia bot

Using the post-receive hook every commit is pushed to the ciabot using .git/hooks/ciabot.pl. This uses the xml-rpc interface for cia.

[edit] mirroring to github

OpenSimulator Github mirror

The opensim-mirror id is used to mirror the OpenSim git repo out to GitHub every 10 minutes. This is one directional and done under cron.

[edit] Unit Testing

Jenkin OpenSimulator Dashboard

Jenkins test results are announced on the #opensim-dev IRC channel

Orginally, test builds were done on Bamboo. Then Bamoboo was replaced by Panda, a custom unit test tool developed by Sean Dague specifically for OpenSimulator. Today, OpenSimulator has retired Panda and is using Jenkins for test builds. More details can be found on the Jenkins page.

[edit] viewgit

Viewgit is configured in /var/www/opensimulator.org/viewgit, and provides web access to the git repo.

[edit] svn tracking mirror

The OpenSim-mirror id also uses a very crude script to attempt to create an svn mirror of the git repo. This is quite an error-prone program, as svn seems to love to remove permissions on executables. Hopefully, this will eventually be retired.

[edit] Backups

This is a two step process:

  • A nightly cronjob that dumps the mysql databases to /var/dbbackups
  • A backuppc process running at Sean Dague's house that sshes in nightly and backs up
    • /home
    • /etc
    • /var/www
    • /var/svn
    • /var/dbbackups

Backuppc has aged backups, so we've got dailies for a week, weeklies for a month, monthlies for 4 months, and every 4 months for 16 months. After the data losses on the previous box, good backups were a must.

[edit] Todo

  • Mediawiki Upgrade (and switch to https protocol?)
  • Find a new backup solution
  • Upgrade mantis to latest stable
Personal tools
General
About This Wiki