The openMosix HOWTO Live free() or die() Kris Buytaert and Others Revision History Minor Fixes Revision v1.0.3 18 june 2004 Revision v1.0.2 29 july 2003 RPM Build Major updates Revision v1.0.1 19 july 2003 Revision v1.0 09 july 2003 Revision v1.0 11 may 2003 Minor updates At last Revision v1.0 RC 1 07 may 2003 Major Cleaning Revision v0.95 04 april 2003 This is an inserted line Revision v0.94 25 february 2003 Replaced ClumpOS by PlumpOS Patches by Mirko Caserta This is an inserted line This is an inserted line Extra features and fixes Revision v0.92 21 january 2003 Revision v0.91 27 september 2002 Revision v0.90 03 september 2002 Revision v0.71 26 August 2002 Spleling Fexis Revision v0.70 22 August 2002 Revision v0.50 6 July 2002 Revision v0.20 5 July 2002 First openMosix HOWTO Latest Mosix HOWTO (for now) Revision v0.17 28 June 2002 Revision v0.15 13 March 2002 Revision v0.13 18 Feb 2002 This is an inserted line Revision ALPHA 0.03 09 October 2001 it." (Benjamin Disraeli) "The best way to become acquainted with a subject is to write a book about ----------------------------------------------------------------------------- I. Introduction Table of Contents 1. Introduction 2. So what is openMosix Anyway ? II. Installing openMosix 3. Requirements and Planning 4. Distribution specific installations 5. Autodiscovery 6. PlumpOS 7. Cluster Installation III. Administrating openMosix This is an inserted line 8. Administrating openMosix 9. Tuning Mosix This is an inserted line 10. openMosixview 11. Other openMosix related Programs 12. Common Problems 13. Hints and Tips 14. (stress)Testing your openMosix installation IV. Running Applications on openMosix 15. Improving Compiling Performance 16. Imaging with openMosix This is a replaced line This is a replaced line 18. Getting started with openMosix internals V. openMosix Development VI. FAQ 19. the openMosix FAQ 20. PlumpOS FAQ This is an inserted line A. How to produce openMosix's Kernel RPM files A.1. How to produce openMosix's Kernel RPM files B. More Info B.1. irc B.2. Further Reading B.4. Links B.5. Mailing List B.3. Translations C. Credits D. GNU Free Documentation License 1. APPLICABILITY AND DEFINITIONS 2. VERBATIM COPYING 3. COPYING IN QUANTITY 0. PREAMBLE 5. COMBINING DOCUMENTS 4. MODIFICATIONS 6. COLLECTIONS OF DOCUMENTS 7. AGGREGATION WITH INDEPENDENT WORKS 9. TERMINATION 10. FUTURE REVISIONS OF THIS LICENSE How to use this License for your documents Index List of Tables 2-1. Pros of openMosix 2-2. Cons of openMosix 4-1. Other Directories 8-1. Changing /proc/hpc parameters 8-3. Writing a 1 to the following files /proc/hpc/decay/ 8-2. /proc/hpc/admin/ 8-4. Informations about the other nodes 8-5. Additional Informations about local processes 8-6. more detailed 8-7. extra options for mosrun 10-1. how to start I. Introduction Table of Contents 1. Introduction 1.1. openMosix HOWTO 1.3. Disclaimer 1.2. Introduction 1.5. New versions of this document 1.4. Distribution policy 1.6. Feedback 2. So what is openMosix Anyway ? 2.1. A very, very brief introduction to clustering 2.2. The story so far 2.3. openMosix in action: An example 2.4. Components 2.5. openMosix Test Drive 2.6. Pros of openMosix 2.7. Cons of openMosix ----------------------------------------------------------------------------- Chapter 1. Introduction 1.1. openMosix HOWTO In the beginning there was Mosix, then came openMosix, in my opinion a more interesting project. Not only from a technical point of view but also due to the more correct license. I made the decision to focus this HOWTO on openMosix rather than on Mosix, mainly based on the fact that openMosix has a bigger userbase. (Moshe Bar states that about 97% of the old Mosix community has switched over to openMosix.) (20020705) Given the above, lots of information might be valuable to both users of Mosix and openMosix. I decided to split the HOWTO. The latest release of the Mosix HOWTO, containing info about both Mosix and OpenMosix will be 0.20 My intention is to focus on the openMosix HOWTO, however not neglecting the Mosix users. More info on http:// howto.ipng.be/Mosix-HOWTO/ ----------------------------------------------------------------------------- This is a replaced line 1.2. Introduction This document gives a brief description of openMosix, a software package the way, some background to parallel processing is given, as well as a brief introduction to programs that make special use of openMosix's capabilities. The HOWTO expands on the documentation as it provides more background This is a replaced line Since the creation of this HOWTO some people of the Mosix team created This is an inserted line openMosix (more info later), initially both openMosix and Mosix were discussed in this HOWTO. Although lots of information might be valuable to both users of Mosix and openMosix. I decided to split the HOWTO. The latest will be 0.20 and can be found on http://howto.ipng.be/Mosix-HOWTO/Mosix-HOWTO / Kris Buytaert got involved in this piece of work when Scot Stevenson was looking for somebody to take over the Job: this was during February 2002. While initially we discussed both Mosix and openMosix, this version of the HOWTO now mainly focuses on openMosix. Please note that the document often still mentions Mosix where it should read openMosix. You will notice that some of the headings are not as serious as they should be. Scot had planned to write the HOWTO in a slightly lighter style, as the world (and even the part of the world with a burping penguin as a mascot) is full of technical literature that is deadly. Therefore some parts still have these comments. ----------------------------------------------------------------------------- 1.3. Disclaimer Use the information in this document at your own risk. I disavow potential This is a replaced line and/or other content of this document is entirely at your own risk. All copyrights are owned by their respective owners, unless specified otherwise. Use of a term in this document should not be regarded affecting the validity of any trademark or service mark. openMosix is Copyright (c) by Moshe Bar. Mosix is Copyright (c) by Amnon Barak. Linux is a Registered Trademark of Linus Torvalds. openMosix is licensed under version 2 of the GNU General Public License as published by the Free Software Foundation. Naming of particular products or brands should not be seen as endorsements. You are strongly recommended to take a backup of your system before major This is an inserted line installation and backups at regular intervals. ----------------------------------------------------------------------------- This is an inserted line 1.4. Distribution policy Copyright (c) 2002 by Kris Buytaert and Scot W. Stevenson. This document may This is an inserted line 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in the appendix entitled "GNU Free Documentation License". ----------------------------------------------------------------------------- 1.5. New versions of this document Official New versions of this document can be found on the web pages of the Linux Documentation Project Drafts and Beta versions will be available on usually be discussed on the openMosix Mailing Lists. See the openMosix for howto.ipng.be in the appropriate sub folder. Changes to this document will details. This is a replaced line Currently this HOWTO is being maintained by Kris Buytaert. Please do send remarks and updates for the howto to him. If you have a technical question about openMosix/Mosix itself, please post them on the more appropriate mailing list. ----------------------------------------------------------------------------- Chapter 2. So what is openMosix Anyway ? This is a replaced line Most of the time, your computer is bored. Start a program like xload or top load is not even hitting the 1.0 mark. If you have two or more computers, This is a replaced line chances are that at any given time, at least one of them is doing nothing. Unfortunately, when you really do need CPU power - during a C++ compile, or encoding Ogg Vorbis music files - you need a lot of it at once. The idea behind clustering is to spread these loads among all available computers, using the resources that are free on other machines. The basic unit of a cluster is a single computer, also called a "node". Clusters can grow in size - they "scale" - by adding more machines. A cluster as a whole will be more powerful the faster the individual computers and the faster their connection speeds are. In addition, the operating system of the cluster must make the best use of the available hardware in response to changing conditions. This becomes more of a challenge if the cluster is composed of different hardware types (a "heterogeneous" cluster), if the configuration of the cluster changes unpredictably (machines joining and leaving the cluster), and the loads cannot be predicted ahead of time. 2.1.1. A very, very brief introduction to clustering ----------------------------------------------------------------------------- 2.1.1.1. HPC vs Fail-over vs Load-balancing Basically there are 3 types of clusters, Fail-over, Load-balancing and HIGH Performance Computing, The most deployed ones are probably the Failover cluster and the Load-balancing Cluster.   * Fail-over Clusters consist of 2 or more network connected computers with a separate heartbeat connection between the 2 hosts. The Heartbeat connection between the 2 machines is being used to monitor whether all the services are still in use: as soon as a service on one machine breaks   * With load-balancing clusters the concept is that when a request for say a web-server comes in, the cluster checks which machine is the least busy and then sends the request to that machine. Actually most of the times a Load-balancing cluster is also a Fail-over cluster but with the extra load balancing functionality and often with more nodes.   * The last variation of clustering is the High Performance Computing that require extreme performance what they need. Beowulfs have been developed especially to give research facilities the computing speed they need. These kind of clusters also have some load-balancing features; they try to spread different processes to more machines in order to gain performance. But what it mainly comes down to in this situation is that a process is being parallelized and that routines that can be ran separately will be spread on different machines instead of having to wait till they get done one after another. Most common known examples of loadbalancing and failover clusters are webfarms, databases or firewalls. People want to have a 99,99999% uptime for their services, the internet is open 24/24 7/7/ 365/365 not unlike in the old days when you could shut down your server when the office closed. for their environments, as long as they can use the maximum power of their machines when they need it. ----------------------------------------------------------------------------- 2.1.1.2. Supercomputers vs. clusters Traditionally Supercomputers have only been built by a selected number of machine had to have a huge budget available for its Supercomputer. Lots of universities could not afford the costs of a Supercomputer by themselves, cluster was born when people first tried to spread different jobs over more more common hardware available to everybody, results similar to real Supercomputers were only to be dreamed of during the first years, but as the PC platform developed further, the performance gap between a Supercomputer and a cluster of multiple personal computers became smaller. ----------------------------------------------------------------------------- 2.1.1.3. Cluster models [(N)UMA, PVM/MPI] This is an inserted line This is an inserted line There are different ways of doing parallel processing: (N)UMA, DSM, PVM and MPI are all different kinds of Parallel Processing schemes. Some of them are implemented in hardware, others in software, others in both. This is a replaced line to the memory where they can execute their code. In the Linux kernel there is This is an inserted line a NUMA implementation that varies the memory access times for different regions of memory. It then is the kernel's task to use the memory that is the closest to the CPU it is using. distributed memory. hardware , the concept is to provide an abstraction layer for physically PVM and MPI are the tools that are most commonly being used when people talk about GNU/Linux based Beowulfs. MPI stands for Message Passing Interface. It is the open standard specification for message passing libraries. MPICH is one of the most used implementations of MPI. Next to MPICH you also can find LAM, another implementation of MPI based on the free reference implementation of the This is a replaced line PVM or Parallel Virtual Machine is another cousin of MPI that is also quite often being used as a tool to create a Beowulf. PVM lives in user space so no special kernel modifications are required: basically each user with enough rights can run PVM. ----------------------------------------------------------------------------- 2.1.1.4. openMosix's role The openMosix software package turns networked computers running GNU/Linux into a cluster. It automatically balances the load between different nodes of the cluster, and nodes can join or leave the running cluster without disruption of the service. The load is spread out among nodes according to their connection and CPU speeds. Since openMosix is part of the kernel and maintains full compatibility with Linux, a user's programs, files, and other resources will all work as before without any further changes. The casual user will not notice the difference between a Linux and an openMosix system. To her, the whole cluster will function as one (fast) GNU/Linux system. openMosix is a Linux-kernel patch which provides full compatibility with algorithm transparently migrates processes to other cluster members. The This is a replaced line advantage is a better load-sharing between the nodes. The cluster itself tries to optimize utilization at any time (of course the sysadmin can affect the automatic load-balancing by manual configuration during runtime). a BIG SMP-system with as many processors as available cluster-nodes (of This transparent process-migration feature makes the whole cluster look like course multiplied with X for X-processor systems such as dual/quad systems and so on). openMosix also provides a powerful optimized File System (oMFS) for HPC-applications, which unlike NFS provides cache, time stamp and link consistency. ----------------------------------------------------------------------------- 2.2. The story so far 2.2.1. Historical Development Rumours say that Mosix comes from Moshe Unix. Initially Mosix started out as an application running on BSD/OS 3.0. Announcing MO6 for BSD/OS 3.0 Oren Laadan (orenl@cs.huji.ac.il) Tue, 9 Sep 1997 19:50:12 +0300 (IDT) Hi: We are pleased to announce the availability of MO6 Version 3.0 Release 1.04 (beta-4) - compatible with BSD/OS 3.0, patch level MO6 is a 6 processor version of the MOSIX multicomputer enhancements LAN, you can experience truly multi-computing environment by using the MO6 enhancements. The MO6 Distribution -------------------- MO6 is available either in "source" or "binary" distribution. It is installed as a patch to BSD/OS, using an interactive installation script. MO6 is available at http://www.cnds.jhu.edu/mirrors/mosix/ or at our site: http://www.cs.huji.ac.il/mosix/ This is an inserted line Main highlights of the current release: -------------------------------------- This is a replaced line - Improved installation procedure. - Enhanced migration control. This is a replaced line - More user utilities. - More documentation and new man pages. Please send feedback and comments to mosix@cs.huji.ac.il. ------------------- GNU/Linux was chosen as a development platform for the 7th incarnation in 1999. Early 1999 Mosix M06 Beta was released for Linux 2.2.1 At the end of 2001 and early 2002 openMosix, the open version of Mosix was born (more in the next paragraph). This is an inserted line ----------------------------------------------------------------------------- 2.2.2. openMosix openMosix is in addition to whatever you find at mosix.org and in full appreciation and respect for Prof. Barak's leadership in the outstanding Mosix project. Moshe Bar has been involved for a number of years with the Mosix project (www.mosix.com) and was co-project manager of the Mosix project and general manager of the commercial Mosix company. This is an inserted line After a difference of opinions on the commercial future of Mosix, he has started a new clustering company - Qlusters, Inc. - and Prof. Barak has decided not to participate for the moment in this venture (although he did seriously consider joining) and held long running negotiations with This is a replaced line project. Because there is a significant user base out there (about 1000 installations world-wide), Moshe Bar has decided to continue the development and support of the Mosix project under a new name: openMosix and under the project is Copyright 2002 by Amnon Barak. All the new code is Copyright 2002 full GPL2 license. Whatever code in openMosix comes from the old Mosix by Moshe Bar. There could (and will) be significant changes in the architecture of the node-discovery and new user-land tools are being discussed in the openMosix future openMosix versions. New concepts about auto-configuration, mailing lists. Most of these new functionalities are already implemented while some of them, such as DSM (Distributed Shared Memory) are still being To approach standardization and future compatibility the proc-interface has etc/hpc.map. More recently the standard for the config file has been set to This is an inserted line be located in /etc/openmosix.map (this is in fact the first config file the / This is an inserted line tools for openMosix are already available on the web-page of the project. etc/init.d/openmosix script will look for). Adapted command-line user-space The openmosix.map config file can be replaced with a node-auto-discovery system which is called omdiscd (openMosix auto DISCovery Daemon) about which we will discuss later. openmosix.sourceforge.net) working together around the world. The main goal openMosix is supported by various competent people (see of the project is to create a standardized clustering-environment for all kinds of HPC-applications. This is a replaced line with a CVS tree and mailing-lists for developers as well as users. ----------------------------------------------------------------------------- 2.2.3. Current state Like most active Open Source programs, openMosix's rate of change tends to outstrip the followers' ability to keep the documentation up to date. As I write this part in February 2003 openMosix 2.4.20 is available and This is an inserted line openMosix Userland Tools v0.2.4 are available, including the new This is a replaced line This is a replaced line For a more recent state of development please take a look at the openMosix website ----------------------------------------------------------------------------- 2.2.4. Which applications work It is almost impossible to give a list off all the applications that work This is a replaced line with openMosix. The community however tries to keep track of the applications that migrate and the ones who don't. ----------------------------------------------------------------------------- This is an inserted line 2.3. openMosix in action: An example openMosix clusters can take various forms. To demonstrate this, let's assume you are a student and share a dorm room with a rich computer science guy, with whom you have linked computers to form an openMosix cluster. Let's Vorbis for your private use, which is legal in your country. Your roommate is This is an inserted line also assume you are currently converting music files from your CDs to Ogg working on a project in C++ that he says will bring World Peace. However, at This is a replaced line computer is idle. So when you start a program like bladeenc to convert Bach's .... from .wav to .ogg format, the openMosix routines on your machine compare the load on This is a replaced line This is an inserted line or click your commands as you would if you were on a standalone machine. All faster, and the response time doesn't go down. Now while you're still typing ...., your roommate comes back, mumbling something about red chili peppers in cafeteria food. He resumes his tests, using a program called 'pmake', a version of 'make' optimized for parallel even starts to send subprocesses to your machine to balance the load. execution. Whatever he's doing, it uses up so much CPU time that openMosix This setup is called *single-pool*: all computers are used as a single cluster. The advantage/disadvantage of this is that your computer is part of yours too. the pool: your stuff will run on other computers, but their stuff will run on ----------------------------------------------------------------------------- This is an inserted line 2.4. Components 2.4.1. Process migration This is an inserted line With openMosix you can start a process on one machine and find out it actually runs on another machine in the cluster. Each process has its own Unique Home Node (UHN) where it gets created. Migration means that a process is splitted in 2 parts, a user part and a system part. The user part will be moved to a remote node while the system part will stay on the UHN. This system-part is sometimes called the deputy process: this process takes care of resolving most of the system calls. openMosix takes care of the communication between these 2 processes. ----------------------------------------------------------------------------- 2.4.2. The openMosix File System (oMFS) oMFS is a feature of openMosix which allows you to access remote filesystems in a cluster as if they were locally mounted. The filesystems of your other ----------------------------------------------------------------------------- home on node 3 on each machine in /mfs/3/home. 2.4.3. Direct File System Access (DFSA) Both Mosix and openMosix provide a cluster-wide file-system (MFS) with the DFSA-option (Direct File-System Access). It provides access to all local and remote file-systems of the nodes in a Mosix or openMosix cluster. 2.5. openMosix Test Drive account to his online openMosix cluster service, which users can use to test and experiment openMosix with. In support of openMosix, Major Chai Mee Joon is giving OM users a free trial The availability of this online openMosix cluster service will help both new users overcome the initial openMosix configuration issues, and also provides higher computing power to openMosix users who are developing or porting their applications. ----------------------------------------------------------------------------- Please send an email to for a trial account. This is an inserted line 2.6. Pros of openMosix Table 2-1. Pros of openMosix +---------------------------------------------------------------------------+ |No extra packages are required. | |No code changes to your application are required. | +---------------------------------------------------------------------------+ This is an inserted line |Simple to install/configure. | |On a Red-Hat based system/distro, installing openMosix is as simple as | +---------------------------------------------------------------------------+ This is a replaced line +---------------------------------------------------------------------------+ This is a replaced line +---------------------------------------------------------------------------+ |Well integrated with openAFS. | +---------------------------------------------------------------------------+ |Port to IA-64 as well as AMD-64 is underway. | +---------------------------------------------------------------------------+ |oMFS has been improved much since plain MFS. | +---------------------------------------------------------------------------+ |It is a clustering platform with more than 10 products based on it: | |openMosixView, openMosixWebView, openMosixApplet, RxLinux, PlumpOS, | +---------------------------------------------------------------------------+ |openMosix is a product developed by the users themselves so it's more close| |K12LTSP, LTSP and many others. | |to the user by definition. | |tools via multicast messaging. | +---------------------------------------------------------------------------+ |Aliases for hosts with multiple interfaces. | +---------------------------------------------------------------------------+ |undesirable). | +---------------------------------------------------------------------------+ |Cluster Mask allows to specify to which nodes a given process can migrate. | +---------------------------------------------------------------------------+ ----------------------------------------------------------------------------- 2.7. Cons of openMosix Table 2-2. Cons of openMosix This is a replaced line |Kernel dependent. | +---------------------------------------------------------------------------+ This is a replaced line +---------------------------------------------------------------------------+ |late march 2003). | |There are issues with Multiple Threads not gaining performance. | +---------------------------------------------------------------------------+ |You won't gain performance when running one single process such as your web| |browser on an openMosix Cluster: the process won't spread itself over the | |cluster. Except of course your process will migrate to a more performant | |machine. | +---------------------------------------------------------------------------+ II. Installing openMosix This is an inserted line Table of Contents This is a replaced line 3.2. Hardware Setup Guidelines 3.5. Classrooms 3.4. Planning your Cluster 4.1. Installing openMosix 4.2. Before getting openMosix 4.4. openMosix General Instructions 4.3. Getting openMosix 4.5. Red Hat and openMosix This is a replaced line 4.7. Debian and openMosix 4.8. openMosix and Gentoo 4.9. Other distributions 5. Autodiscovery 5.1. Easy Configuration 5.3. Troubleshooting autodiscovery 5.2. Compiling auto-discovering 6. PlumpOS 6.1. What Plump/OS is 6.2. How does it work? 6.3. Requirements 6.4. Getting Started This is an inserted line 7. Cluster Installation 7.1. Cluster Installations 7.2. DSH, Distributed Shell This is an inserted line ----------------------------------------------------------------------------- Chapter 3. Requirements and Planning This is a replaced line This is a replaced line This is an inserted line Installing a basic cluster requires at least 2 network connected machines, either using a cross-cable between the two network cards or using a switch or hub (a switch is much better than a hub though and only costs a few bucks This is an inserted line more). Of course the faster your network-cards the easier you will get better performance for your cluster. These days Fast Ethernet (100 Mbps) is standard; putting multiple ports in a machine isn't that difficult, but make sure to connect them through other physical networks in order to gain the speed you want. Gigabit Ethernet is getting cheaper every day now but I suggest that you don't rush to the shop spending your money before you have actually tested your setup with multiple 100Mbit cards and noticed that you really do need the extra network capacity. Next to putting a Gigabit card you might also want to try bonding different 100Mbit cards together. An even cheaper alternative can be found in Firewire, as discussed in this paper ----------------------------------------------------------------------------- 3.2. Hardware Setup Guidelines Setting up a big cluster requires some thinking to be done: where are you going to put the machines? Not under a table somewhere or in the middle of your office I hope! It's ok if you just want to do some small tests, but if you are planning to deploy a N node cluster you will have to make sure that the environment that will hold these machines is capable of doing so. This is a replaced line I'm talking about preparing one or more 19" racks to host the machines, configuring the appropriate network topology, either straight, single connected or even a 1 to 1 cross connected network between all your nodes. You will also need to make sure that there is enough power to support such a range of machines, that your air-conditioning system supports the load and that in case of power-failure your UPS can cleanly shut down all the required systems. You might want to invest in a KVM (Keyboard, Video, Mouse) Switch in This is a replaced line This is an inserted line investment, make sure that you can always easily access the different nodes, you never know when you have to replace a CPU fan or an hard-disk of a machine in trouble. If that means that you have to unload a stack of machines to reach the bottom one, hence shutting down your cluster, you are in trouble. ----------------------------------------------------------------------------- 3.3. Software requirements This is an inserted line The systems we plan to use will need a basic Linux installation of your really matter which one. What does matter is that the kernel is at least on choice: Red Hat, SuSe, Debian, Gentoo or any another distribution: it doesn't 2.4 level, and that your network-cards are configured correctly; next to that This is a replaced line This is an inserted line ----------------------------------------------------------------------------- 3.4. Planning your Cluster This is a replaced line This is a replaced line set of (personal) workstations, you have different options that will have their advantages and disadvantages. as a single cluster: each machine is a part of the cluster and can   * In a Single-pool configuration all the servers and workstations are used migrate processes to each other existing node. This of course makes your workstation a part of the pool.   * In an environment that is called a Server-pool, servers are a part of the cluster while workstations aren't part of it, they don't even have openMosix kernel. If you want to run applications on the cluster you will need to specifically log on to these servers. However your workstation will also stay clean and no remote processes will migrate to it.   * A third alternative is called an Adaptive-pool configuration: here This is a replaced line you log out in the evening, a script tells the workstation to join the cluster and start crunching numbers. This way your machine is being used while you don't need it. If you need the resources of the machine again just run the openmosix stop script and your processes will stay away from the cluster and vice-versa. Practically this means that you will change the role of your machine by using mosctl. This is an inserted line This is a replaced line ----------------------------------------------------------------------------- 3.5. Classrooms openMosix cluster at night, you'll have to consider training your end users not to pull the power switch of those machines when they want to use them again. More recent machines support automatic shutdowns when hitting the power button, but with older machines you might loose some data when this This is an inserted line This is a replaced line This is an inserted line Chapter 4. Distribution specific installations 4.1. Installing openMosix This chapter deals with installing openMosix on different distributions. It won't be an exhaustive list of all the possible combinations. However throughout the chapter you should find enough information on installing openMosix in your environment. Techniques for installing multiple machines with openMosix will be discussed in one of the next chapters. ----------------------------------------------------------------------------- 4.2. Before getting openMosix This is a replaced line First of all, you must understand that openMosix is made up of a kernel patch and some user-space tools. The kernel patch is needed to make the kernel capable of talking to other openMosix-enabled machines on the network. If you download openMosix as a binary package (such as an rpm file), you don't even and compiled with the most common default options and modules for you. need to take care about the kernel patch because the kernel has been patched The user-space tools are needed in order to make an effective use of an openMosix-enabled kernel. They are needed to start/stop the migration daemon, the openMosix File System, to migrate jobs to certain nodes and other tasks which are usually accomplished with the help our good old friend: the command line interface. About binary packages: the same as in the kernel patch goes for the user-space tools: if you install an rpm you don't need to care about compiling them or configuring anything; just let them install and run. That's Once you get to the download page (which we'll talk about in a second), This is an inserted line all. Really :) you'll need to get two distinct parts: the kernel and the user-space tools. You can either download two binary packages or get the kernel patch plus the scheme: openMosix-x.y.z-w where x.y.z is the version of the vanilla Linux Kernel against which the patch should be applied and w is the patch revision for that particular kernel release. For the precompiled kernel binaries, please refer to the README-openMosix-kernel.txt file you'll find in the download page. This file also contains updated info about manually compiling a kernel. This is an inserted line openmosix-tools. We use the terms user-space tools, userspace-tools and openmosix-tools interchangeably. Updated info about precompiled binaries and manually compiling the tools are also provided in the README-openmosix-tools.txt file. Please note that since version 0.3 of the openmosix-tools, the openmosix.map file is deprecated and the use of the autodiscovery daemon is highly encouraged since it tends to make your life ----------------------------------------------------------------------------- 4.3. Getting openMosix /project/showfiles.php?group_id=46729. You can either choose the binary (even in rpm) compiled for UP or SMP or download the source code. You will need both the kernel patch or binaries and the userland tools. Alternatively you can get the CVS version: cvs -d:pserver:anonymous@cvs.openmosix.sourceforge.net:/cvsroot/openmosix login cvs -z3 -d:pserver:anonymous@cvs.openmosix.sourceforge.net:/cvsroot/openmosix co linux-openmosix cvs -z3 -d:pserver:anonymous@cvs.openmosix.sourceforge.net:/cvsroot/openmosix co userspace-tools At the password prompt, just type enter since you're doing an anonymous This is a replaced line ----------------------------------------------------------------------------- might not be the easiest way to install openMosix ;-) 4.4. openMosix General Instructions 4.4.1. Kernel Compilation Always use pure vanilla kernel-sources from http://www.kernel.org/ to compile an openMosix kernel! Please be kind enough to download the kernel using a mirror near to you and always try and download patches to the latest kernel sources you do have instead of downloading the whole thing. This is going to be much appreciated by the Linux community and will greatly increase your geeky Karma ;-) Be sure to use the right openMosix patch depending on the kernel-version. At the moment I write this, the latest 2.4 kernel is 2.4.20 so you should download the openMosix-2.4.20-x.gz patch, where the "x" stands for the patch revision (ie: the greater the revision number, the most recent it is). Do not use the kernel that comes with any Linux-distribution: it won't work. These kernel sources get heavily patched by the distribution-makers so, applying the openMosix patch to such a kernel is going to fail for sure! Been there, done that: trust me ;-) Download the actual version of the openMosix patch and move it in your kernel-source directory (e.g. /usr/src/linux-2.4.20). If your kernel-source creation of a symbolic link to "/usr/src/linux-[version_number]" is required. directory is other than "/usr/src/linux-[version_number]" at least the in your home directory, apply the patch using (guess what?) the patch utility: mv /root/openMosix-2.4.20-2.gz /usr/src/linux-2.4.20 cd /usr/src/linux-2.4.20 zcat openMosix-2.4.20-2.gz | patch -Np1 In the rare case you don't have "zcat" on your system, do: This is an inserted line mv /root/openMosix-2.4.20-2.gz /usr/src/linux-2.4.20 cd /usr/src/linux-2.4.20 gunzip openMosix-2.4.20-2.gz cat openMosix-2.4.20-2 | patch -Np1 This is a replaced line mv /root/openMosix-2.4.20-2.gz /usr/src/linux-2.4.20 cd /usr/src/linux-2.4.20 gunzip openMosix-2.4.20-2.gz patch -Np1 < openMosix-2.4.20-2 kernel-sources. If you feel adventurous enough, enable the openMosix related options in the kernel-configuration file, e.g. This is an inserted line ... # CONFIG_MOSIX_TOPOLOGY is not set CONFIG_MOSIX=y CONFIG_MOSIX_UDB=y # CONFIG_MOSIX_CHEAT_MIGSELF is not set CONFIG_MOSIX_WEEEEEEEEE=y CONFIG_MOSIX_DIAG=y CONFIG_MOSIX_SECUREPORTS=y CONFIG_MOSIX_DISCLOSURE=3 This is an inserted line CONFIG_QKERNEL_EXT=y CONFIG_MOSIX_DFSA=y CONFIG_MOSIX_FS=y This is a replaced line CONFIG_QOS_JID=y ... However, it's going to be pretty much easier if you configure the above options using one of the Linux-kernel configuration tools: The above means you have to choose one of "config", "menuconfig", and This is an inserted line "xconfig". It's a matter of taste. By the way, "config" is going to work on any system; "menuconfig" needs the curses libraries installed while "xconfig" make config | menuconfig | xconfig needs an installed X-window environment plus the TCL/TK libraries and interpreters. Now compile it with: make dep bzImage modules modules_install After compilation install the new kernel with the openMosix options within you boot-loader; e.g. insert an entry for the new kernel in /etc/lilo.conf and run lilo after that. ----------------------------------------------------------------------------- 4.4.2. Syntax of the /etc/openmosix.map file This is an inserted line Before starting openMosix, there has to be an /etc/openmosix.map configuration file which must be the same on each node. This is an inserted line The standard is now /etc/openmosix.map, /etc/mosix.map and /etc/hpc.map are old standards, but the CVS-version of the tools is backwards compatible and order). looks for /etc/openmosix.map, /etc/mosix.map and /etc/hpc.map (in that This is an inserted line The openmosix.map file contains three space separated fields: openMosix-Node_ID IP-Address(or hostname) Range-size An example openmosix.map file could look like this: 1 node1 1 2 node2 1 3 node3 1 4 node4 1 or 2 192.168.1.2 1 1 192.168.1.1 1 3 192.168.1.3 1 4 192.168.1.4 1 or with the help of the range-size both of the above examples equal to: openMosix "counts-up" the last byte of the ip-address of the node according 1 192.168.1.1 4 to its openMosix-Node_ID. Of course, if you use a range-size greater than 1 you have to use ip-addresses instead of hostnames. If a node has more than one network-interface it can be configured with the ALIAS option in the range-size field (which equals to setting the range-size This is an inserted line to 0) e.g. 1 192.168.1.1 1 2 192.168.1.2 1 3 192.168.1.3 1 4 192.168.1.4 1 Here the node with the openMosix-Node_ID 4 has two network-interfaces 4 192.168.10.10 ALIAS (192.168.1.4 + 192.168.10.10) which are both visible to openMosix. Always be sure to run the same openMosix version AND configuration on each of your Cluster's nodes! setpe -w -f /etc/openmosix.map Start openMosix with the "setpe" utility on each node : Execute this command (which will be described later on in this HOWTO) on every node in your openMosix cluster. Alternatively, you can grab the "openmosix" script which can be found in the scripts directory of the userspace-tools, copy it to the /etc/init.d /etc/init.d/openmosix stop /etc/init.d/openmosix restart Installation is finished now: the cluster is up and running :) ----------------------------------------------------------------------------- 4.4.3. oMFS First of all, the CONFIG_MOSIX_FS option in the kernel configuration has to be enabled. If the current kernel was compiled without this option, then recompilation with this option enabled is required. This is a replaced line This is a replaced line openldap. The CONFIG_MOSIX_DFSA option in the kernel is optional but of This is a replaced line has to be an additional fstab-entry on each node's /etc/fstab. This is a replaced line mfs_mnt /mfs mfs dfsa=1 0 0 in order to have DFSA disabled: mfs_mnt /mfs mfs dfsa=0 0 0 the syntax of this fstab-entry is: [device_name] [mount_point] mfs defaults 0 0 After mounting the /mfs mount-point on each node, each node's file-system is going to be accessible through the /mfs/[openMosix-Node_ID]/ directories. With the help of some symbolic links all cluster-nodes can access the same data e.g. /work on node1 on node2 : ln -s /mfs/1/work /work This is an inserted line on node3 : ln -s /mfs/1/work /work ... This is an inserted line The following special files are excluded from the oMFS:   * the /proc directory This is an inserted line   * special files which are not regular-files, directories or symbolic links This is an inserted line (e.g. /dev/hda1) Creating links like: ln -s /mfs/1/mfs/1/usr or ln -s /mfs/1/mfs/3/usr is invalid. This is an inserted line The following system calls are supported without sending the migrated process This is an inserted line (which executes this call on its home (remote) node) going back to its home node: This is an inserted line read, readv, write, writev, readahead, lseek, llseek, open, creat, close, dup, dup2, fcntl/fcntl64, getdents, getdents64, old_readdir, fsync, fdatasync, chdir, fchdir, getcwd, stat, stat64, newstat, lstat, lstat64, This is a replaced line This is an inserted line ftruncate64, chmod, chown, chown16, lchown, lchown16, fchmod, fchown, This is a replaced line rename This is a replaced line work: This is an inserted line   * different mfs/dfsa configuration on the cluster-nodes This is a replaced line This is a replaced line   * pathnames that leave the DFSA-filesystem   * when the process which executes the system-call is being traced system-call Next to the /mfs/1/ /mfs/2/ and so on files you will find some other directories as well. This is an inserted line Table 4-1. Other Directories +--------+------------------------------------------------------------------+ |/mfs/ |The current node where your process runs | |here | | +--------+------------------------------------------------------------------+ |/mfs/ |Your home node | |home | | +--------+------------------------------------------------------------------+ |magic |"open" with the "O_CREAT" option) - otherwise, the last node on | | |which an oMFS magical file was successfully created (this is very | | |useful for creating temporary-files, then immediately unlinking | | |them) | +--------+------------------------------------------------------------------+ |/mfs/ |The node on which the process last issued a successful "execve" | |lastexec|system-call. | +--------+------------------------------------------------------------------+ |/mfs/ |The node you selected by either your process itself or one of its | |selected|ancestor's (before forking this process), writing a number into "/| | |proc/self/selected". | +--------+------------------------------------------------------------------+ Note that these magic files are all ``per process''. That is their content is dependent upon which process opens them. A last not about openMFS is that there are versions around that return faultive results when you run "df" on those filesystems. Don't be surpised if you suddenlty have about 1.3 TB available on those systems. ----------------------------------------------------------------------------- 4.5. Red Hat and openMosix If you are running a RedHat 7.2, 7.3 or 8.0 version, this is probably the This is a replaced line This is an inserted line RPMs from sourceforge. They have precompiled kernels (as I write this 2.4.20) that work seamlessly: I have tested them on several machines including Laptops with PCMCIA cards and Servers with SCSI disks. If you are a grub user, the kernel rpm even modifies your grub.conf. So all you have to do is This is an inserted line install 2 RPMs: rpm -Uvh openmosix-kernel-2.4.20-openmosix2.i686.rpm openmosix-tools-0.2.4-1.i386.rpm This is a replaced line daemon (omdiscd). Since this seems to be a problem for lots of people, let's go with another example. Say you have 3 machines: 192.168.10.220, 192.168.10.78 and 192.168.10.84. Your openmosix.map will look like this. This is a replaced line # openMosix CONFIGURATION This is a replaced line # # Each line should contain 3 fields, mapping IP addresses to openMosix node-numbers: # 1) first openMosix node-number in range. # 2) IP address of the above node (or node-name from /etc/hosts). # 3) number of nodes in this range. # # Example: 10 machines with IP 192.168.1.50 - 192.168.1.59 # 1 192.168.1.50 10 # # ============================ This is an inserted line 1 192.168.10.220 1 2 192.168.10.78 1 3 192.168.10.84 1 This is a replaced line Most RedHat installations have one extra thing to fix. You often get the following error: [root@inspon root]# /etc/init.d/openmosix start setpe: the supplied table is well-formatted, Initializing openMosix... This means that your hostname is not listed in /etc/hosts with the same ip as but my IP address (127.0.0.1) is not there! in your openmosix.map. You might have a machine called omosix1.localhost.org in your hostfile listed as 127.0.0.1 omosix1.localhost.org localhost If you modify your /etc/hosts to look like below, openMosix will have less troubles starting up. 192.168.10.78 omosix1.localhost.org 127.0.0.1 localhost [root@inspon root]# /etc/init.d/openmosix start [root@inspon root]# /etc/init.d/openmosix status Initializing openMosix... This is openMosix node #2 Network protocol: 2 (AF_INET) openMosix range 1-1 begins at 192.168.10.220 openMosix range 2-2 begins at inspon.localhost.be openMosix range 3-3 begins at 192.168.10.84 Total configured: 3 This is an inserted line If you would like to use more bleeding edge patches, you can always opt for the src rpm and run rpmbuild --rebuild on it. This will install the source for you and create an initial config file. From there you can go further applying patches to openMosix A tutorial on how to build your own openMosix RPM's can be found in the Appendixes. feel free to drop the author a note and help him keeping this information updated. ----------------------------------------------------------------------------- 4.6. Suse and openMosix Although the RPMs are being built on a RedHat based environment, you can use most of them on other RPM based systems. Suse however has /sbin/mk_initrd as a link to /sbin/mkinitrd, which makes rpms before release 20-2 fail. Newer version should have a fix for this. ----------------------------------------------------------------------------- This is an inserted line 4.7. Debian and openMosix Installing openMosix ``the Debian way'' can be easily done as described This is an inserted line below. The first step consists in downloading the packages from the net. I had to This is an inserted line needed: http://packages.debian.org/unstable/net/openmosix.html, http:// packages.debian.org/unstable/misc/kernel-package.html, http:// packages.debian.org/unstable/net/kernel-patch-openmosix.html, http:// packages.debian.org/unstable/devel/kernel-source-2.4.19.html. You can also apt-get install them ;). The next part is making the kernel openMosix capable. Basically, the procedure to follow is: cd /usr/src apt-get install kernel-source-2.4.19 kernel-package \ openmosix kernel-patch-openmosix tar vxjf kernel-source-2.4.19.tar.bz2 ln -s /usr/src/kernel-source-2.4.19 /usr/src/linux make menuconfig ../kernel-patches/i386/apply/openmosix make-kpkg kernel_image modules_image cd .. dpkg -i kernel-image-*-openmosix-*.deb You now will need to edit your /etc/openmosix.map. Please follow the instructions given in the ``Syntax of /etc/openmosix.map'' part of this HOWTO. This is an inserted line After rebooting with this kernel and a configured /etc/openmosix.map, you should then have a cluster of openMosix machines that talk to each-other and This is a replaced line You can test that by running the following small script: awk 'BEGIN {for(i=0;i<10000;i++)for(j=0;j<10000;j++);}' a couple of times, and monitor its behaviour with "mosmon" where you will see that it spreads the load between the different nodes. We also setup openMosixView on the Debian machine: In order to be able to actually use openMosixView you will need to run it apt-get install openmosixview from a user who can log in to the different nodes as root. We suggest you set this up using ssh. Please note that there is a difference between the ssh and ssh2 implementations. If you do have an identity.pub file, ssh will check authorized_keys, while if you do have an id_dsa.pub you will need This is a replaced line openMosixView gives you a nice interface that shows the load of different machines and gives you the possibility to migrate processes manually. A detailed discussion of openMosixView can be found elsewhere in this document. ----------------------------------------------------------------------------- 4.8. openMosix and Gentoo First Install Gentoo Linux This is an inserted line Then, install openMosix: type "emerge sys-apps/openmosix-user", which will install an openMosix kernel source tree in /usr/src/linux along with the openMosix userland tools. This is an inserted line Michael Imhof, aka tantive, keeps Gentoo current for the latest openMosix version. Daniel Robbins, the President/CEO of Gentoo Technologies, Inc. and the creator of Gentoo Linux, wrote the artitles we use as our Introduction to ----------------------------------------------------------------------------- 4.9. Other distributions Based on the explanations above you should be able to install openMosix on most other Linux platforms. ----------------------------------------------------------------------------- Chapter 5. Autodiscovery 5.1. Easy Configuration The auto-discovery daemon (omdiscd) provides a way to automatically configure an openMosix cluster hence eliminating the need of a /etc/mosix.map or similar manual configurations. Auto-discovery uses multicast packages to notify other nodes that it is an openMosix node. This way adding an extra your machine and it will join the cluster. node to your mosix cluster means that you just have to start the omdiscd on However there are some small requirements, Like with any openMosix cluster , This is an inserted line you need to have networking configured correctly. mainly the routing. Without Otherwise omdiscd will exit with an error like. Aug 31 20:41:49 localhost omdiscd[1290]: Unable to determine address of default interface. This may happen because there is no default route configured. Without a default route, an interface must be: Network is unreachable Aug 31 20:41:49 localhost omdiscd[1290]: Unable to initialize network. Exiting. An example of a correct routing is below [root@localhost log]# route -n Kernel IP routing table 10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth0 Destination Gateway Genmask Flags Metric Ref Use Iface 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo 0.0.0.0 10.0.0.99 0.0.0.0 UG 0 0 0 eth0 Basically from now on everything will get easier. Just start omdiscd And have a look at your logfiles you should see something similar to this Sep 2 10:00:49 oscar0 kernel: openMosix configuration changed: This is openMosix #2780 (of 6 configured) Sep 2 10:00:49 oscar0 kernel: openMosix #2780 is at IP address 192.168.10.220 Sep 2 10:00:49 oscar0 kernel: openMosix #2638 is at IP address 192.168.10.78 Sep 2 10:00:49 oscar0 kernel: openMosix #2646 is at IP address 192.168.10.86 Sep 2 10:00:49 oscar0 kernel: openMosix #2634 is at IP address 192.168.10.74 Sep 2 10:00:49 oscar0 kernel: openMosix #2627 is at IP address 192.168.10.67 Congratulations , your openMosix cluster is now working. omdiscd has some other options that you can use. You can either run omdiscd as a daemon (default) or in the foreground where output goes to the screen (standard output) omdiscd -n . An interface can be specified with the -i option. Now lets still have a short look at the other tool , it's showmap. This tool will show you the newly auto generated openMosix map. [root@oscar0 root]# showmap My Node-Id: 0x0adc Base Node-Id Address Count ------------ ---------------- ----- 0x0adc 192.168.10.220 1 0x0a4e 192.168.10.78 1 0x0a56 192.168.10.86 1 0x0a43 192.168.10.67 1 0x0a4a 192.168.10.74 1 Auto-discovery has some other features not listed here such as a routing mechanism for clusters with more than one network. More detailed information This is a replaced line tree. More recent versions of the openMosix rc scripts will first verify wether an autoconfiguration. ----------------------------------------------------------------------------- 5.2. Compiling auto-discovering modification to openmosix.c. One of the first lines will be If you are compiling autodiscovery from source you will need to make a small #define ALPHA You will need to put this in comment. If you want to have some more logging available to you you should edit main.c to show log_set_debug % make clean (DEBUG_TRACE_ALL); (somewhere around line 84) now run % make ----------------------------------------------------------------------------- 5.3. Troubleshooting autodiscovery example a node might not see multicast traffic from other nodes. This has Sometimes however autodiscovery does not function as you would like, for occurred with some PCMCIA ethernet drivers. One solution is to place the Aug 31 20:45:58 localhost kernel: openMosix configuration changed: This is openMosix #98 (of 1 configured) This is a replaced line openMosix Aug 31 20:45:58 localhost kernel: Received an unauthorized information request from 10.0.0.99 What you should to then is try to force your NIC into promiscuous and/ or This is a replaced line ifconfig ethx promisc This is an inserted line or This is an inserted line ifconfig ethx multicast You might also want to run This is an inserted line tcpdump -i eth0 ether multicast which will have the same effect but you will now also be able to see the packages yourself. On some Layer 3 switches you other configs might be required. An openMosix user found out that on his Switch Summit48Si (Extreme Networks) he had to run disable ipmcforwarding (to deactivate the routing of multicast paquets) disable igmp snooping before he was the different omdiscd's were able to see eachother, other switches might require similar configs. Aug 31 22:14:43 inspon omdiscd[1422]: Simulated notification to activate openMosix This is a replaced line My Node-Id: 0x0063 Base Node-Id Address Count ------------ ---------------- ----- 0x0063 10.0.0.99 1 This is an inserted line This is a replaced line [root@inspon root]# If you see the simulated you have probably forgotten to put the This is a replaced line in comment. I have also noticed that autodiscovery does not work with FireWire based network cards. ----------------------------------------------------------------------------- Chapter 6. PlumpOS 6.1. What Plump/OS is PlumpOS is a CD-based GNU/Linux/openMosix mini-distribution designed to allow users to quickly, or temporarily, add nodes to an openMosix cluster; as I This is a replaced line download. This chapter is a quick hack up by Peter Willis of a very similar chapter contributed by Jean-David Marrow (who is the author of Clump/OS and inspirer ----------------------------------------------------------------------------- 6.2. How does it work? This is an inserted line Check your motherboard's manual to see how to set up this. Upon booting up you will receive a prompt to boot PlumpOS using a specific kernel; several ones have been provided for you but, you also have the option of providing your own kernel + modules. This will be explained later in the ``Getting for you to use; simply type the name of one (and optionally some kernel Started'' section. The boot menu should tell you all the kernels available arguments) and press return/enter. At boot-time, PlumpOS will auto-probe for network cards and, if any gets up omdiscd on each interface a DHCP lease is received on. Currently this has only been tested in PCs with 1 network card so YMMV. detected, it'll try and configure them via DHCP. If successful, it will fire It works for myself, but may not work for you; if you experience difficulties, please email me with as much information about your system as possible -- after you have investigated the problem. (Check the sourceforge This is an inserted line much detailed information as possible. The openMosix-general and openMosix-devel lists are NOT for PlumpOS problems, they are for openMosix-specific problems, and if you can't tell the difference just send ----------------------------------------------------------------------------- it to the PlumpOS list or my contact email). 6.3. Requirements you already have a running openMosix cluster -- or perhaps only a single openMosix node -- from which you will be initiating jobs. All machines in the cluster must conform to the following requirements:   * PlumpOS Machine(s) 586+ CPU   * bootable CD-ROM drive This is an inserted line   * Network Interface Card   * Between 32M and 128M of RAM. Because of some oddities in the openMosix kernels at the time of this writing, simply passing the size of the ramdisk is not working so one has to actually use a ramdisk pre-made for a specific size. There should be several ramdisks avaliable at PlumpOS mirrors which can be placed in the disks' root directory on the actual cdrom ISO (or rootdisk/disks/ if you're inside the PlumpOS package directory).   * Master Machine(s) GNU/Linux/openMosix kernel (same version as all the PlumpOS kernel you are booting on the Child/Slave PlumpOS Machine(s)) This is a replaced line DHCP, you can still manually configure your system: simply go to each the supplied networking tools. Using DHCP is highly recommended however, PlumpOS machine and enter in your desired configuration information with and will greatly simplify your life in the long run. This is an inserted line This is an inserted line The following network modules are present in most if not all of the supplied kernels' modules tarball (in /kernels/KERNELNAME/modules.tgz), although not all support auto-probing; if you don't see support for your card in this list, then PlumpOS will not work for you. 3c501.o, 3c503.o, 3c505.o, 3c507.o, 3c509.o, 3c515.o, 3c59x.o, 8139cp.o, 8139too.o, 82596.o, ac3200.o, acenic.o, aironet4500_card.o, aironet4500_core.o, aironet4500_proc.o, arlan-proc.o, arlan.o, at1700.o, bsd_comp.o, cs89x0.o, de4x5.o, depca.o, dgrs.o, dl2k.o, dmfe.o, dummy.o, e100 This is a replaced line eth16i.o, ewrk3.o, fealnx.o, hamachi.o, hp-plus.o, hp.o, hp100.o, lance.o, lp486e.o, mii.o, natsemi.o, ne.o, ne2k-pci.o, ni5010.o, ni52.o, ni65.o, This is an inserted line ppp_synctty.o, pppoe.o, pppox.o, sis900.o, sk98lin/sk98lin.o, slhc.o, ns83820.o, pcmcia, pcnet32.o, ppp_async.o, ppp_deflate.o, ppp_generic.o, This is a replaced line tc35815.o, tg3.o, tlan.o, tokenring/{3c359.o abyss.o ibmtr.o lanstreamer.o olympic.o smctr.o tms380tr.o tmsisa.o tmspci.o}, tulip/tulip.o, via-rhine.o, wavelan.o, wd.o, winbond-840.o, wireless/{airo.o airo_cs.o hermes.o orinoco.o orinoco_cs.o orinoco_pci.o orinoco_plx.o}, yellowfin.o Please also note that PlumpOS may not work on a laptop: it definitely doesn't support PCMCIA cards (yet), and will probably not configure openMosix properly if your machine contains multiple connected Ethernet adapters. This is a temporary limitation of the configuration scripts, and should be ----------------------------------------------------------------------------- 6.4. Getting Started You can download the latest PlumpOS package under the terms of the GPL, without warranty of any kind, from any PlumpOS mirror. Upon downloading, unpack the archive $ tar -xvzf plumpos-6.9-rc1.tar.gz and enter the new directory "plumpos-6.9-rc1/". Now you have several options as to how you go about setting up PlumpOS. First let's familiarize with the directory structure here. There should be 3 This is an inserted line This is an inserted line directories:   * rootdisk, which contains the layout of the iso to be created; This is an inserted line   * scripts, which contains small programs used to help in the creation of the iso; This is a replaced line   * final, which will contain the ISO generated by the whole process. This is a replaced line done configuring your system. If you intend on using your own kernel with PlumpOS, there are several steps This is a replaced line often blamed DoS 8.3 character format and only contain letters and numbers directory in the rootdisk/kernels/ directory that will be in the old and (and a single '.'). In that directory you should put a file named bzImage which is your kernel and a file named modules.tgz which is a gzipped tarball of your kernel's modules (with the relative path of lib/modules/ so it can be and config file for your kernel. When this is done you may simply run the This is a replaced line install program and it will detect and install your kernel(s) for use with This is a replaced line Now, say you want to include a 3rd party add-on package. This is done very in your package (relative to "/") and put these packages into the rootdisk/ simply: create a gzipped tarball containing all the files you want to include packages/ directory. Then edit the file rootdisk/packages/list and add the path relative to / on the ramdisk where the package can be located (in other words for a package named "openssl.tar.gz", add the line "/cdrom/packages/ openssl.tar.gz" to the file rootdisk/packages/list). Optionally you can use a line such as "cdrom:openssl.tar.gz" which will automatically search the releases this will be useful for things like nfs and boot floppies, so for cdrom's packages directory for the package "openssl.tar.gz". In future now don't worry about it ;) ----------------------------------------------------------------------------- Chapter 7. Cluster Installation 7.1. Cluster Installations This chapter does not deal with installing openMosix as such, it does however This is an inserted line deal with installing multiple machines with openMosix. Automated or semi automated mass installs. ----------------------------------------------------------------------------- 7.2. DSH, Distributed Shell At the time of this writing (May 2003) DSH's most current release is available from http://www.netfort.gr.jp/~dancer/software/downloads/ More info on the package can be found on http://www.netfort.gr.jp/~dancer/software/ dsh.html The latest version available for download is 0.23.6 You will need both libdshconfig-0.20.8.tar.gz and dsh-0.23.5.tar.gz Start with installing libdshconfig ./configure make make install Repeat the process for the dsh package. Say we have a small cluster with a couple of nodes. To make life easier we want type each command once but have it executed on each node. You then have to create a file in $HOME/.dsh/group/clusterwname that lists the ip's of your This is an inserted line This is an inserted line cluster. eg. [root@inspon root]# cat .dsh/group/mosix 192.168.10.84 As an example we run ls on each of these machines We use -g to use the mosix group (this way you can create subsets of a group with different configurations) [root@inspon root]# dsh -r ssh -g mosix ls 192.168.10.84: anaconda-ks.cfg 192.168.10.84: id_rsa.pub 192.168.10.84: install.log 192.168.10.84: install.log.syslog 192.168.10.84: openmosix-tools-0.2.0-1.i386.rpm 192.168.10.84: openmosix-kernel-2.4.17-openmosix1.i686.rpm 192.168.10.220: id_dsa.pub 192.168.10.220: anaconda-ks.cfg 192.168.10.220: id_rsa.pub This is an inserted line This is an inserted line 192.168.10.220: openmosix-kernel-2.4.17-openmosix1.i686.rpm 192.168.10.220: openmosix-tools-0.2.0-1.i386.rpm 192.168.10.220: oscar-1.2.1rh72 192.168.10.220: oscar-1.2.1rh72.tar.gz Note that neither of the machines ask for a password. This is because we have set up RSA authentication between the different accounts. If you want to run commands with multiple parameters you will either have to put the command between quotes. [root@inspon root]# dsh -r ssh -g mosix "uname -a" Wed May 29 14:32:28 CEST 2002 i686 unknown 192.168.10.84: Linux omosix2.office.be.stone-it.com 2.4.17-openmosix1 #1 192.168.10.220: Linux oscar0 2.4.17-openmosix1 #1 Wed May 29 14:32:28 CEST 2002 i686 unknown or use the -c -- option. Both give basically the same output. This is a replaced line 2002 i686 unknown 192.168.10.220: Linux oscar0 2.4.17-openmosix1 #1 Wed May 29 14:32:28 CEST Wed May 29 14:32:28 CEST 2002 i686 unknown 192.168.10.84: Linux omosix2.office.be.stone-it.com 2.4.17-openmosix1 #1 III. Administrating openMosix Table of Contents 8. Administrating openMosix 8.1. Basic Administration 8.2. Configuration 8.3. the userspace-tools This is an inserted line 8.4. Cluster Mask 9. Tuning Mosix 9.1. Introduction 9.2. Creating a "Master" node This is a replaced line 9.4. Channel Bonding Made Easy 9.5. Updatedb 9.6. openMosix and FireWire This is a replaced line 10.1. Introduction 10.2. openMosixview vs Mosixview This is a replaced line 10.5. openMosixprocs 10.6. openMosixcollector 10.7. openMosixanalyzer 10.8. openMosixmigmon 10.9. openmosixview FAQ This is an inserted line This is a replaced line 11. Other openMosix related Programs 11.2. openMosixView 11.1. Introduction 11.3. openMosixapplet 11.4. wmonload 11.5. openMosixWebView 12. Common Problems 12.1. Introduction This is an inserted line 12.2. My processes won't migrate 12.3. I don't see all my nodes 12.4. I often get errors: No such process 12.5. DFSA ? MFS ? 12.6. Python Troubles 13. Hints and Tips 13.2. Choosing your processes 13.1. Locked Processes 13.3. Java and openMosix 13.4. openMosix and Hyperthreading 13.5. openMosix and Firewalls This is a replaced line 14.1. A small Test Script 14.3. the openMosix stress-test This is a replaced line This is a replaced line Chapter 8. Administrating openMosix ----------------------------------------------------------------------------- 8.1. Basic Administration openMosix provides the advantage of process migration to HPC-applications. This is a replaced line openMosix-user-space-tools or the /proc/hpc interface which will be now This is an inserted line described in detail. Up till openMosix version 2.4.16 the /proc interface was named /proc/mosix ! ----------------------------------------------------------------------------- 8.2. Configuration The values in the flat files in the /proc/hpc/admin directory presenting the current configuration of the cluster. Also the administrator can write its own values into these files to change the configuration during runtime, e.g. Table 8-1. Changing /proc/hpc parameters +------------------------------+---------------------------------------+ +------------------------------+---------------------------------------+ |echo 1 > /proc/hpc/admin/block|blocks the arrival of remote processes | |echo 1 > /proc/hpc/admin/bring|bring all migrated processes home | +------------------------------+---------------------------------------+ ... Table 8-2. /proc/hpc/admin/ +------------+--------------+-----------------------------------------------+ |(binary |config |the main configuration file (written by the | |files) | |setpe util) | +------------+--------------+-----------------------------------------------+ |(flat files)|block |allow/forbid arrival of remote processes | This is a replaced line |  |bring |bring home all migrated processes | +------------+--------------+-----------------------------------------------+ |  |dfsalinks |list of current symbolic dfsa-links | +------------+--------------+-----------------------------------------------+ +------------+--------------+-----------------------------------------------+ |  |expel |sending guest processes home | |  |gateways |maximum number of gateways | +------------+--------------+-----------------------------------------------+ |  |lstay |local processes should stay | +------------+--------------+-----------------------------------------------+ |  |mospe |contains the openMosix node id | |  |nomfs |disables/enables MFS | +------------+--------------+-----------------------------------------------+ +------------+--------------+-----------------------------------------------+ |  |overheads |for tuning | +------------+--------------+-----------------------------------------------+ |  |quiet |stop collecting load-load-balancing | | | |informations | +------------+--------------+-----------------------------------------------+ | | |load-balancing | |  |decay-interval|interval for collecting informations about | +------------+--------------+-----------------------------------------------+ |  |slow-decay |default 975 | +------------+--------------+-----------------------------------------------+ |  |fast-decay |default 926 | +------------+--------------+-----------------------------------------------+ |  |speed |speed relative to PIII/1GHz) | |  |stay |enables/disables automatic process migration | +------------+--------------+-----------------------------------------------+ +------------+--------------+-----------------------------------------------+ This is an inserted line This is a replaced line |clear |clears the decay statistics | +------+----------------------------------------------+ +------+----------------------------------------------+ |cpujob|tells openMosix that the process is cpu-bound | +------+----------------------------------------------+ |iojob |tells openMosix that the process is io-bound | +------+----------------------------------------------+ |slow |tells openMosix to decay its statistics slow | +------+----------------------------------------------+ This is a replaced line This is an inserted line Table 8-4. Informations about the other nodes +------------------------------------+--------------------------------------+ +------------------------------------+--------------------------------------+ |/proc/hpc/nodes/[openMosix_ID]/CPUs |how many CPU's the node has | +------------------------------------+--------------------------------------+ |/proc/hpc/nodes/[openMosix_ID]/load |the openMosix load of this node | |/proc/hpc/nodes/[openMosix_ID]/mem |available memory as openMosix believes| +------------------------------------+--------------------------------------+ |/proc/hpc/nodes/[openMosix_ID]/rmem |available memory as Linux believes | +------------------------------------+--------------------------------------+ |/proc/hpc/nodes/[openMosix_ID]/speed|speed of the node relative to PIII/ | This is a replaced line |/proc/hpc/nodes/[openMosix_ID]/ |status of the node | +------------------------------------+--------------------------------------+ |status | | This is an inserted line |/proc/hpc/nodes/[openMosix_ID]/tmem |available memory | +------------------------------------+--------------------------------------+ +------------------------------------+--------------------------------------+ This is a replaced line +------------------------------------+--------------------------------------+ Table 8-5. Additional Informations about local processes +-------------------------+----------------------------------------------+ |/proc/[PID]/cantmove |reason why a process cannot be migrated | +-------------------------+----------------------------------------------+ |/proc/[PID]/goto |to which node the process should migrate | +-------------------------+----------------------------------------------+ |/proc/[PID]/lock |if a process is locked to its home node | |/proc/[PID]/nmigs |how many times the process migrated | +-------------------------+----------------------------------------------+ |/proc/[PID]/where |where the process is currently being computed | |/proc/[PID]/migrate |same as goto remote processes | +-------------------------+----------------------------------------------+ |/proc/hpc/remote/from |the home node of the process | +-------------------------+----------------------------------------------+ |/proc/hpc/remote/identity|additional informations about the process | This is an inserted line +-------------------------+----------------------------------------------+ |/proc/hpc/remote/statm |memory statistic of the process | +-------------------------+----------------------------------------------+ +-------------------------+----------------------------------------------+ ----------------------------------------------------------------------------- 8.3. the userspace-tools This is a replaced line These following tools are providing easy administration to openMosix migrate -send a migrate request to a process This is an inserted line syntax: migrate [PID] [openMosix_ID] mon -is a ncurses-based terminal monitor several informations about the current status are displayed in bar-charts mosctl -is the openMosix main configuration utility This is an inserted line syntax: mosctl [stay|nostay] [lstay|nolstay] [quiet|noquiet] [block|noblock] [nomfs|mfs] [expel|bring] [gettune|getyard|getdecay] mosctl whois [openMosix_ID|IP-address|hostname] mosctl [getload|getspeed|status|isup|getmem|getfree|getutil] [openMosix_ID] This is a replaced line mosctl setspeed interger-value mosctl setdecay interval [slow fast] Table 8-6. more detailed +--------+------------------------------------------------------------------+ |stay |no automatic process migration | +--------+------------------------------------------------------------------+ |nostay |automatic process migration (default) | +--------+------------------------------------------------------------------+ |lstay |local processes should stay | This is a replaced line |nolstay |local processes could migrate | |block |block arriving of guest processes | |noblock |allow arriving of guest processes | +--------+------------------------------------------------------------------+ This is an inserted line +--------+------------------------------------------------------------------+ |quiet |disable gathering of load-balancing informations | |noquiet |enable gathering of load-balancing informations | +--------+------------------------------------------------------------------+ |nomfs |disables MFS | +--------+------------------------------------------------------------------+ +--------+------------------------------------------------------------------+ |mfs |enables MFS | +--------+------------------------------------------------------------------+ This is a replaced line |bring |bring all migrated processes home | +--------+------------------------------------------------------------------+ +--------+------------------------------------------------------------------+ This is an inserted line |gettune |shows the current overhead parameter | +--------+------------------------------------------------------------------+ |getyard |shows the current used Yardstick | +--------+------------------------------------------------------------------+ |getdecay|shows the current decay parameter | +--------+------------------------------------------------------------------+ |whois |resolves openMosix-ID, ip-addresses and hostnames of the cluster | +--------+------------------------------------------------------------------+ |getload |display the (openMosix-) load | |getspeed|shows the (openMosix-) speed | +--------+------------------------------------------------------------------+ |status |displays the current status and configuration | +--------+------------------------------------------------------------------+ |isup |is a node up or down (openMosix kind of ping) | +--------+------------------------------------------------------------------+ This is an inserted line |getmem |shows logical free memory | +--------+------------------------------------------------------------------+ |getfree |shows physical free mem | +--------+------------------------------------------------------------------+ |getutil |display utilization | |setyard |sets a new Yardstick-value | +--------+------------------------------------------------------------------+ This is a replaced line |setspeed|sets a new (openMosix-) speed value | +--------+------------------------------------------------------------------+ This is an inserted line |setdecay|sets a new decay-interval | +--------+------------------------------------------------------------------+ This is a replaced line This is a replaced line mosrun [-h|openMosix_ID| list_of_openMosix_IDs] command [arguments] The mosrun command can be executed with several more commandline options. To ease this up there are several preconfigured run-scripts for executing jobs with a special (openMosix) configuration. Table 8-7. extra options for mosrun +---------+-----------------------------------------------------------------+ |nomig |runs a command which process(es) won't migrate | This is an inserted line +---------+-----------------------------------------------------------------+ |runhome |executes a command locked to its home node | |runon |runs a command which will be directly migrated and locked to a | +---------+-----------------------------------------------------------------+ | |node | +---------+-----------------------------------------------------------------+ |cpujob |tells the openMosix cluster that this is a cpu-bound process | +---------+-----------------------------------------------------------------+ |iojob |tells the openMosix cluster that this is a io-bound process | +---------+-----------------------------------------------------------------+ |nodecay |executes a command and tells the cluster not to refresh the | | |load-balancing statistics | +---------+-----------------------------------------------------------------+ |slowdecay|executes a command with a slow decay interval for collecting | | |load-balancing statistics | |fastdecay|executes a command with a fast decay interval for collecting | +---------+-----------------------------------------------------------------+ This is a replaced line +---------+-----------------------------------------------------------------+ setpe -manual node configuration utility syntax: setpe -w -f [hpc_map] setpe -r [-f [hpc_map]] -w reads the openMosix configuration from a file (typically /etc/hpc.map) -r writes the current openMosix configuration to a file (typically /etc/hpc.map) -off turns the current openMosix configuration off tune openMosix calibration and optimizations utility. (for further informations review the tune-man page) Additional to the /proc interface and the commandline-openMosix utilities (which are using the /proc interface) there is a patched "ps" and "top" available (they are called "mps" and "mtop") which displays also the openMosix-node ID on a column. This is useful for finding out where a specific process is currently being computed. This actually summarised the command line tools, but have a look at openMosixview which is a GUI for the most common administration tasks, and which ill be discussed in a future chapter. ----------------------------------------------------------------------------- 8.4. Cluster Mask This is an inserted line (by Moshe Bar) Several people have asked for a feature in openMosix which allows to specifiy to which nodes a given process and it's children can migrate and to which This is a replaced line This is an inserted line Simone Ettore has just committed a new patch to the CVS which allows you to This is an inserted line do just that. Here is how it works:   * /proc/[pid]/migfilter enable/disable the capability of filter migration.   * /proc/[pid]/mignodes is a bit-list of nodes. The bit position of a node is calculated as 2^(PE-1). PE is node number.   * /proc/[pid]/migpolicy is the policy of the filtering: 0=DENY: the process can migrate in all nodes except when the relative bit on mignodes is 1 1= mignodes is 1 This is a replaced line We are shortly going to release also a simple user-land tool to set the node openMosix 2.4.20-3. ----------------------------------------------------------------------------- Chapter 9. Tuning Mosix 9.1. Introduction these parts will get replaced by relevant openMosix parts, however some Some of the parts below are still from the old Mosix Howto, as time passes things are still the same , but your mileage may vary. ----------------------------------------------------------------------------- 9.2. Creating a "Master" node Although openMosix architcture does not require a master node as such, you might want to have a head node from where you launch processes, this might be a multihomed node from where users log in to your cluster. You want to configure your machine to make processes migrate away You have to trick the node in thinking it is the slowest node around and it'd better migrate all it's processes to the faster nodes. You will have to make it "slow" with : This is a replaced line where n should be much lower than the speed of the other nodes Processes will move/migrate away fast. You can get the speed of a node with : mosctl getspeed ----------------------------------------------------------------------------- 9.3. Optimizing Mosix This is an inserted line Editorial Comment: To be checked with openMosix versions Login a normal terminal as root. Type setpe -r which, if everything went right, will give you a listing of your /etc/ mosix.map. If things did not go right, try setpe -w -f /etc/mosix.map to set up your node. Then, type cat /proc/$$/lock to see if your child processes are locked in your mode (1) or can migrate (0). If for some reason you find your processes are locked, you can change this with echo 0 > /proc/$$/lock until you fix the problem. Repeat the whole configuration scheme for a second computer. The programs tune_kernel and prep_tune that Mosix uses to calibrate the individual nodes do not work with the SuSE distribution. However, you can Mosix installed down to single user mode by typing fake it. First, bring the computer you want to tune and another computer with init 1 On both machines, run the following commands: as root. All other computers on the network should be shutdown if possible. This is an inserted line /etc/init.d/network start /etc/init.d/mosix start echo 1 > /proc/mosix/admin/quiet This fakes prep_tune and the first parts of tune_kernel. Note that if you have a laptop with a pcmcia network card, you will have to run /etc/init.d/pcmcia start instead of "/etc/init.d/network start". On the computer you want to tune, run take a while - if you have a dog, this might be the time to go on that long, long walk you've always promised him. tune_kernel will create a program This is a replaced line recompile the kernel). Repeat the tuning procedure for each computer. Reboot, This is an inserted line enjoy Mosix, and don't forget to brag to your friends about your new cluster. ----------------------------------------------------------------------------- This is an inserted line Channel bonding is actually horrible easy. This may explain the lack of This is a replaced line the applications. All machines on a subnet must be bonded the same way. Bonded and non-bonded machine really don't talk well to each other. This is a replaced line Channel bonding needs at least two physical sub-nets but can have more (Currently I have a tri-bonded cluster). To enable bonding you need to either This is a replaced line kernel code, as of 2.4.x is it a standard option of the kernel. The NIC's are This is a replaced line first card of the bond. 'ifenslave' is used to initialize the remaining cards in the bonded connection. 'ifenslave' can be locate in the linux/ file. The basic format for use is ifenslave ... Documentation/network/ directory. It will need to be compiled as it is a .c Channel bonded networks can connect to standard networks via a router or bridge that supports channel bonding( I just use an extra NIC and port-forwarding in the head node). ----------------------------------------------------------------------------- 9.5. Updatedb This is a replaced line disable updatedb from indexing this mountpoints. ----------------------------------------------------------------------------- openMosix does gain performance by using another type of network device, as described within the paper about openMosix and FireWire ----------------------------------------------------------------------------- Chapter 10. openMosixview 10.1. Introduction openMosixview is the next version and a complete rewrite of Mosixview. It is a cluster-management GUI for openMosix-cluster and everybody is invited to This is an inserted line download and use it (at your own risk and responsibility). The openMosixview-suite contains 5 useful applications for monitoring and administrating openMosix-cluster. openMosixview the main monitoring+administration application This is a replaced line openMosixcollector collecting daemon which logs cluster+node informations openMosixanalyzer for analyzing the data collected by the openMosixcollector openMosixhistory a process-history for your cluster All parts are accessible from the main application window. The most common openMosix-commands are executable by a few mouse-clicks. An advanced execution dialog helps to start applications on the cluster. "Priority-sliders" for each node simplifying the manual and automatic load-balancing. openMosixview is now adapted to the openMosix-auto-discovery This is an inserted line and gets all configuration-values from the openMosix /proc-interface. ----------------------------------------------------------------------------- 10.2. openMosixview vs Mosixview This is an inserted line openMosixview is fully designed for openMosix cluster only. The Mosixview-website (and all mirrors) will stay as they are but all further This is an inserted line developing will continue with openMosixview located at the new domain www.openmosixview.com If you have: questions, features wanted, problems during installation, comments, exchange of experiences etc. feel free to mail me, Matt Rechenburg or subscribe to the openMosix/Mosixview-mailing-list and mail to the openMosix/Mosixview-mailing-list changes: (to Mosixview 1.1) openMosixview is a complete rewrite "from the scratch" of Mosixview! It has the same functionalities but there are fundamental changes in ALL parts of the openMosixview source-code. It is openMosix auto-discovery) All "buggy" parts are removed or rewritten and it tested with a constantly changing cluster topography (required for the (should ;) run much more stable now. adapted to the openMosix-auto-discovery removed the (buggy) map-file parser not using /etc/mosix.map or any cluster-map file anymore rewrote all parts/functions/methods to a cleaner c++ interface replaced MosixMem+Load with the openMosixanalyzer fixed some smaller bugs in the display ... many more changes This is an inserted line ----------------------------------------------------------------------------- 10.3. Installation Requirements QT library rlogin and rsh (or ssh) to all cluster-nodes without password the openMosix userland-tools mosctl, migrate, runon, iojob, cpujob ... (download them from the www.openmosix.org website) On a RH 8.0 you will need at least the following rpm's qt-3.0.5-17, libmng-1.0.4, XFree86-Mesa-libGLU-4.2.0, glut-3.7 etc ... Documentation about openMosixview There is a full HTML-documentation about openMosixview included in every package. You find the startpage of the documentation in your openMosixview installation directory: openmosixview/ openmosixview/docs/en/index.html The RPM-packages have their installation directories in: /usr/local/ openmosixview ----------------------------------------------------------------------------- e.g.: Download the latest version of openMosixview rpm-package. Then just execute rpm -i openmosixview-1.4.rpm This will install all binaries in /usr/bin To uninstall: ----------------------------------------------------------------------------- 10.3.2. Installation of the source-distribution Download the latest version of openMosixview and unzip+untar the sources and gunzip openmosixview-1.4.tar.gz copy the tarball to e.g. /usr/local/. tar -xvf openmosixview-1.4.tar ----------------------------------------------------------------------------- 10.3.3. Automatic setup-script Just cd to the openmosixview-directory and execute ./setup [your_qt_2.3.x_installation_directory] ----------------------------------------------------------------------------- This is a replaced line 10.3.4. Manual compiling Set the QTDIR-Variable to your actual QT-Distribution, e.g. or setenv QTDIR /usr/lib/qt-2.3.0 (for csh) export QTDIR=/usr/lib/qt-2.3.0 (for bash) ----------------------------------------------------------------------------- This is an inserted line 10.3.5. Hints (from the testers of openMosixview/Mosixview who compiled it on different This is an inserted line linux-distributions, thanks again) Create the link /usr/lib/qt pointing to your QT-2.3.x installation e.g. if QT-2.3.x is installed in /usr/local/ qt-2.3.0 ln -s /usr/local/qt-2.3.0 /usr/lib/qt export QTDIR=/usr/lib/qt (for bash) or After that the rest should work fine: setenv QTDIR /usr/lib/qt (for csh) ./configure make then do the same in the subdirectory openmosixcollector, openmosixanalyzer, openmosixhistory and openmosixviewprocs. Copy all binaries to /usr/bin cp openmosixview/openmosixview /usr/bin cp openmosixviewproc/openmosixviewprocs/mosixviewprocs /usr/bin cp openmosixcollector/openmosixcollector/openmosixcollector /usr/bin cp openmosixanalyzer/openmosixanalyzer/openmosixanalyzer /usr/bin cp openmosixhistory/openmosixhistory/openmosixhistory /usr/bin cp openmosixcollector/openmosixcollector.init /etc/init.d/openmosixcollector or cp openmosixcollector/openmosixcollector.init /etc/rc.d/init.d/openmosixcollector Now copy the openmosixprocs binary on each of your cluster-nodes to /usr/bin/ rcp openmosixprocs/openmosixprocs your_node:/usr/bin/openmosixprocs This is a replaced line openmosixview ----------------------------------------------------------------------------- 10.4. using openMosixview 10.4.1. main application Here is a picture of the main application-window. The functionality is [omview1] openMosixview displays a row with a lamp, a button, a slider, a lcd-number, two progress-bars and some labels for each cluster-member. The lights at the This is an inserted line left are displaying the openMosix-Id and the status of the cluster-node. Red if down, green for available. If you click on a button displaying the ip-address of one node a configuration-dialog will pop up. It shows buttons to execute the most common used "mosctl"-commands. (described later in this HOWTO) With the speed is displayed by the lcd-number. This is an inserted line You can influence the load-balancing of the whole cluster by changing these values. Processes in a openMosix-Cluster are migrating easier to a node with more openMosix-speed than to nodes with less speed. Sure it is not the has. e.g. a cpu-intensive job on a cluster-node which speed is set to the physically speed you can set but it is the speed openMosix "thinks" a node lowest value of the whole cluster will search for a better processor for running on and migrate away easily. The progress bars in the middle gives an overview of the load on each cluster-member. It displays in percent so it does not represent exactly the give an overview. load written to the file /proc/hpc/nodes/x/load (by openMosix), but it should The next progressbar is for the used memory the nodes. It shows the currently used memory in percent from the available memory on the hosts (the label to the right displays the available mem). How many CPUs your cluster This is a replaced line contains a configuration button for "all-nodes". You can configure all nodes This is an inserted line in your cluster similar by this option. This is an inserted line How good the load-balancing works is displayed by the progressbar in the top left. 100% is very good and means that all nodes nearly have the same load. Use the collector- and analyzer-menu to manage the openMosixcollector and open the openMosixanalyzer. This two parts of the openMosixview-application suite are useful for getting an overview of your cluster during a longer period. ----------------------------------------------------------------------------- 10.4.2. the configuration-window This is a replaced line [omview2] The openMosix-configuration of each host can be changed easily now. All local node) so "root" has to "rsh" (or "ssh") to each host in the cluster commands will be executed per "rsh" or "ssh" on the remote hosts (even on the without prompting for a password (it is well described in a Beowulf documentation or on the HOWTO on this page how to configure it). automigration on/off The commands are: exspel yes/no bring/lstay yes/no openMosix start/stop If openMosixprocs is properly installed on the remote cluster-nodes click the "remote proc-box"-button to open openMosixprocs (proc-box) from remote. xhost +hostname will be set and the display will point to your localhost. The openmosixprocs must be copied to e.g. /usr/bin on each host of the cluster) client is executed on the remote also per "rsh" or "ssh". (the binary openMosixprocs is a process-box for managing your programs. It is useful to manage programs started and running local on the remote nodes and is described later in this HOWTO. local hostname in the edit-box below the "remote proc-box". Then If you are logged on your cluster from a remote workstation insert your openMosixprocs will be displayed on your workstation and not on the This is an inserted line cluster-member you are logged on. (maybe you have to set "xhost +clusternode" on your workstation). There is a history in the combo-box so you have to write the hostname only once. ----------------------------------------------------------------------------- 10.4.3. advanced-execution If you want to start jobs on your cluster the "advanced execution"-dialog may This is a replaced line [omview3] Choose a program to start with the "run-prog" button (file-open-icon) and you can specify how and where the job is started by this execution-dialog. ----------------------------------------------------------------------------- 10.4.4. the command-line This is an inserted line top of the window. You can specify additional commandline-arguments in the lineedit-widget on Table 10-1. how to start |-no migration|start a local job which won't migrate | +-------------+-------------------------------------------------------------+ +-------------+-------------------------------------------------------------+ |-run home |start a local job | +-------------+-------------------------------------------------------------+ |-run on |start a job on the node you can choose with the | | |"host-chooser" | |-cpu job |start a computation intensive job on a node (host-chooser) | +-------------+-------------------------------------------------------------+ +-------------+-------------------------------------------------------------+ |-io job |start a io intensive job on a node (host-chooser) | +-------------+-------------------------------------------------------------+ |-no decay |start a job with no decay (host-chooser) | +-------------+-------------------------------------------------------------+ |-slow decay |start a job with slow decay (host-chooser) | +-------------+-------------------------------------------------------------+ |-fast decay |start a job with fast decay (host-chooser) | |-parallel |start a job parallel on some or all node (special | +-------------+-------------------------------------------------------------+ | |host-chooser) | This is a replaced line 10.4.5. the host-chooser This is an inserted line The openMosix-id of the node is also displayed by a lcd-number. Then click execute to start the job. 10.4.6. the parallel host-chooser You can set the first and last node with 2 spinboxes. Then the command will be executed an all nodes from the first node to the last node. You can also inverse this option. ----------------------------------------------------------------------------- This is a replaced line 10.5. openMosixprocs 10.5.1. intro This process-box is really useful for managing the processes running on your cluster. This is a replaced line This is a replaced line The processlist gives an overview what is running where. The second column displays the openMosix-node ID of each process. 0 means local, all other values are remote nodes. Migrated processes are marked with a green icon and non movable processes have a lock. for managing e.g. migrating the process. There are also options to migrate the remote processes away, send SIGSTOP and SIGCONT to it or to "renice" it. This is an inserted line If you click on the "manage procs from remote" button a new window will come up (the remote-procs windows) displaying the process currently migrated to this host. ----------------------------------------------------------------------------- 10.5.2. the migrator-window This dialog will pop up if process from the process box is clicked. [omview5] This is an inserted line The openMosixview-migrator window displays all nodes in your openMosix-cluster. This window is for managing one process (with additional This is an inserted line status-information). By double-clicking on an host from the list the process This is an inserted line will migrate to this host. After a short moment the process-icon for the managed process will be green, which means it is running remote. the process is send to the best available node in your cluster. This This is a replaced line the best speed. With the "kill"-button you can kill the process immediately. To pause a program just click the "SIGSTOP"-button and to continue the "SIGCONT"-button. With the renice-slider below you can renice the current managed process (-20 means very fast, 0 normal and 20 very slow) 10.5.3. managing processes from remote This dialog will pop up if the "manage procs from remote"-button beneath the process-box is clicked [omview6] The TabView displays processes that are migrated to the local host. The procs are coming from other nodes in your cluster and currently computed on the host openMosixview is started on. Similar to the two buttons in the migrator-window the process is send home by the "goto home node"-button and send to the best available node by the "goto best node"-button. ----------------------------------------------------------------------------- 10.6. openMosixcollector The openMosixcollector is a daemon which should/could be started on one cluster-member. It logs the openMosix-load of each node to the directory /tmp /openmosixcollector/* These history log-files analyzed by the openMosixanalyzer (as described later) gives an nonstop overview of the load, memory and processes in your cluster. There is one main log-file called /tmp/ this directory to which the data is written. openmosixcollector/cluster Additional to this there are additional files in openMosixcollector.pid The openMosixcollector-daemon restarts every 12 hours and saves the current This is an inserted line history to /tmp/openmosixcollector[date]/* These backups are done automatically but you can also trigger this manual. There is an option to write a checkpoint to the history. These checkpoints are graphically marked as a blue vertical line if you analyze the history log-files with the openMosixanalyzer. For example you can set a checkpoint This is an inserted line when you start a job on your cluster and another one at the end.. Here is the explanation of the possible commandline-arguments: openmosixcollector -d //starts the collector as a daemon openmosixcollector -k //stops the collector openmosixcollector -n //writes a checkpoint to the history openmosixcollector -r //saves the current history and starts a new one openmosixcollector //print out a short help You can start this daemon with its init-script in /etc/init.d or /etc/rc.d/ init.d. You just have to create a symbolic link to one of the runlevels for automatic startup. How to analyze the created logfiles is described in the openMosixanalyzer-section. ----------------------------------------------------------------------------- 10.7. openMosixanalyzer This is a replaced line This picture shows the graphical Load-overview in the openMosixanalyzer This is a replaced line This is a replaced line This is a replaced line With the openMosixanalyzer you can have a non-stop openMosix-history of your cluster. The history log-files created by openMosixcollector are displayed in a graphically way so that you have a long-time overview what happened and happens on your cluster. The openMosixanalyzer can analyze the current "online" logfiles but you can also open older backups of your tmp/openmosixcollector/* (the backups in /tmp/openmosixcollector[date]/*) and openMosixcollector history logs by the filemenu. The logfiles are placed in / you have to open only the main history file "cluster" to take a look at older load-informations. (the [date] in the backup directories for the log-files is the date the history is saved) The start time is displayed on the top and you have a full-day view in the openMosixanalyzer (12 h). If you are using the openMosixanalyzer for looking at "online"-logfiles (current history) you can enable the "refresh"-checkbox and the view will auto-refresh. The load-lines are normally black. If the load increases to >75 the lines are drawn red. These values are openMosix--informations. The openMosixanalyzer gets these informations from the files /proc/hpc/nodes/ [openMosix ID]/* This is an inserted line The Find-out-button of each nodes calculates several useful statistic values. Clicking it will open a small new window in which you get the average load- and mem values and some more statically and dynamic informations about the specific node or the whole cluster. ----------------------------------------------------------------------------- 10.7.2. statistical informations about a cluster-node [omview9] This is a replaced line If there are checkpoints written to the load-history by the openMosixcollector they are displayed as a vertical blue line. You now can compare the load values at a certain moment much easier. ----------------------------------------------------------------------------- 10.7.3. the memory-overview [omview8] This picture shows the graphical Memory-overview in the openMosixanalyzer With Memory-overview in the openMosixanalyzer you can have a non-stop memory history similar to the Load-overview. The history log-files created by openMosixcollector are displayed in a graphically way so that you have a long-time overview what happened and happens on your cluster. It analyze the openMosixcollector history logs by the filemenu. these informations from the files The displayed values are openMosix-informations. The openMosixanalyzer gets /proc/hpc/nodes/[openMosix-ID]/mem. /proc/hpc/nodes/[openMosix-ID]/rmem. /proc/hpc/nodes/[openMosix-ID]/tmem. This is a replaced line openMosixcollector they are displayed as a vertical blue line. ----------------------------------------------------------------------------- 10.7.4. openMosixhistory [omview10] This is an inserted line displays the processlist from the past collector was started on and you can browse this log-data with node. The openMosixcollector saves the processlist from the host the openMosixhistory. You can easy change the browsing time in openMosixhistory by the time-slider. openMosixhistory can analyze the current "online" logfiles but you can also open older backups of your openMosixcollector history logs by the filemenu. The logfiles are placed in /tmp/openmosixcollector/* (the backups in /tmp/ openmosixcollector[date]/*) and you have to open only the main history file "cluster" to take a look at older load-informations. (the [date] in the backup directories for the log-files is the date the history is saved) The start time is displayed on the top/left and you have a 12 hour view in openMosixhistory. ----------------------------------------------------------------------------- 10.8. openMosixmigmon 10.8.1. General [ommigmon2] The openMosixmigmon is a monitor for migrations in your openMosix-cluster. It displays all your nodes as little penguins sitting in a circle. -> nodes-circle. The main penguin is the node on which openMosixmigmon runs and around this node it shows its processes also in a circle of small black squares. This is an inserted line If a process migrates to one of the nodes the node gets an own process-circle and the process moved from the main process-circle to the remote process-circle. Then the process is marked green and draws a line from its origin to its remote location to visualize the migration. ----------------------------------------------------------------------------- 10.8.2. Tooltips: If you hold your mouse above a process it will show you its PID and This is an inserted line commandline in a small tooltip-window. ----------------------------------------------------------------------------- This is an inserted line 10.8.3. Drag'n Drop! This is a replaced line process and drop them to any of your nodes (those penguins) and the process will move there. If you double-click a process on a remote node it will be send home immediately. This is a replaced line 10.9. openmosixview FAQ 10.9.1. I cannot compile openMosixview on my system? 10.9.2. Can I use openMosixview with SSH? 10.9.4. The openMosixviewprocs/mosixview_client is not working for me! wrong? 10.9.5. Why are the buttons in the openMosixview-configuration dialog not preselected? 10.9.1. I cannot compile openMosixview on my system? At first QT >= 2.3.x is required. The QTDIR -environment variable has to be set to your QT-installation directories like it is well described in the INSTALL- file. In versions < 0.6 you can do a "make clean" and delete the two again because i alway left the binary- and object-files in older versions. If files: /openmosixview/Makefile /openmosixview/config.cache and try to compile you have any other problems post them to the openMosixview-mailinglist (or This is an inserted line This is an inserted line This is a replaced line to ssh to each node in your cluster without password (just like the same with Yes, until version 0.7 there is a built-in SSH-support. You have to be able using RSH this is required) This is an inserted line 10.9.3. I started openMosixview but only the splash-screen appears. What is Do not fork openMosixview in the background with & (e.g. openMosixview &). Maybe you cannot rsh/ssh (depends on what you want to use) as user root without password to each node? Try "rsh hostname" as root. You should not been promped for a password but soon get a login shell. (If you use SSH try "ssh hostname" as root.) You have to be root on the cluster because that is the only way the administrative commands executed by openMosixview requires root-privileges. openMosixview uses "rsh" as the default! If you only have "ssh" installed on your cluster edit (or create) the file /root openMosixview and the last "1" stands for "use ssh instead of rsh". This will /.openMosixview and put "1111" in it. This is the main-configuration file for cause openMosixview to use "ssh" even for the first start. This is a replaced line 10.9.4. The openMosixviewprocs/mosixview_client is not working for me! whith a checkbox) on the remote host. It has to be installed in /usr/bin/ on The openMosixview-client is executed per rsh (or ssh which you can configer This is an inserted line each node. If you use RSH try: "xhost +hostname" "rsh hostname /usr/bin/ This is a replaced line This is a replaced line your_local_host_name:0.0" If this works it will work in openMosixview too. version of openMosixview/Mosixview ? in the mosix.map-parser (which is completly removed in openMosixview !!) (the versions openMosixview 1.2 and Mosixview > 1.0 are stable) 10.9.5. Why are the buttons in the openMosixview-configuration dialog not preselected? too. The problem is to get the information of node. You have to login to each (automigration on/off, blocking on/off......) I want them to be preselected cluster-node because these information are not cluster-wide (to my mind). The status of each node is stored in the /proc/hpc/admin directory of each node. Everybody who knows a good way to get these information easy is invited to mail me. ----------------------------------------------------------------------------- 10.10. openMosixview + ssh: (this HowTo is for SSH2) You can read the reasons why you should use SSH instead of RSH everyday on the newspaper when another script-kiddy hacked into an insecure system/network. So SSH is a good decision at all. freedom x security = constant (from a security newsgroup) That is why it is a bit tricky to configure SSH. SSH is secure even if you use it to login without being prompted for a password. Here is a (one) way to This is an inserted line At first a running secure-shell daemon on the remote site is required. If it [sshd_rpm_packeage_from_your_linux_distribution_cd]) If it is not already running start it with: /etc/init.d/ssh start Now you have to generate a keypair for SSH on your local computer whith ssh-keygen This is a replaced line is longer than a password and may be a whole sentence. The keypair is encrypted with that passphrase and saved in /root/.ssh/identity //your private key /root/.ssh/identity.pub //your public key and Do NOT give your private-key to anybody!!! Now copy the whole content of / root/.ssh/authorized_keys on the remote host. (also copy the content of /root /.ssh/identity.pub to your local /root/.ssh/authorized_keys like you did it root/.ssh/identity.pub (your public-key which should be one long line) into / with the remote-node because openMosixview needed password-less login to the local-node too!) If you ssh to this remote host now you will be prompted for the passphrase of your public-key. Giving the right passphrase should give you a login. What is the advantage right now??? The passphrase is normally a lot longer than a password! The advantage you can get using the ssh-agent. It manages the passphrase during ssh login. The ssh-agent is started now and gives you two environment-variables you ssh-agent should set (if not set already). Type: echo $SSH_AUTH_SOCK and echo $SSH_AGENT_PID to see if they are exported to your shell right now. If not just cut and paste from your terminal. e.g. for the bash-shell: SSH_AUTH_SOCK=/tmp/ssh-XXYqbMRe/agent.1065 SSH_AGENT_PID=1066 export SSH_AGENT_PID example for the csh-shell: setenv SSH_AUTH_SOCK /tmp/ssh-XXYqbMRe/agent.1065 With these variables the remote-sshd-daemon can connect your local ssh-agent by using the socket-file in /tmp (in this example /tmp/ssh-XXYqbMRe/ setenv SSH_AGENT_PID 1066 agent.1065). The ssh-agent can now give the passphrase to the remote host by using this socket (it is of course an encrypted transfer)! You just have to add your public-key to the ssh-agent with the ssh-add This is a replaced line Now you should be able to login using ssh to the remote host without being ssh-add prompted for a passwod! You could (should) add the ssh-agent and ssh-add commands in your login-profile e.g. eval `ssh-agent` ssh-add Now it is started when you login on your local workstation. You have done it! I wish you secure logins now. openMosixview There is a menu-entry which toggles using rsh/ssh with This is an inserted line openMosixview. Just enable this and you can use openMosixview even in possibility for saveing the current config in openMosixview was added in the insecure network-environments. You should also save this configuration (the 0.7 version) because it gets initial data from the slave using rsh or ssh (just like you configured). This is a replaced line not work! (e.g. if you cannot rsh to a slave without being prompted for a password you cannot use openMosixview with RSH; if you cannot ssh to a slave This is a replaced line ----------------------------------------------------------------------------- Chapter 11. Other openMosix related Programs This is an inserted line 11.1. Introduction There are a couple of different applications available to monitor and admin openMosix, we give a short overview of them in this chapter, we won't really go in detail. ----------------------------------------------------------------------------- This is an inserted line 11.2. openMosixView openMosixview is the most used and the best known applet for openMosix administration, you can read more about it in the openMosix adminstration chapter. ----------------------------------------------------------------------------- This is an inserted line 11.3. openMosixapplet The openMosixApplet lets you watch the realtime load of your openMosix applets. The applet uses chart2D to provide a good-lookin' feeling. cluster. It consists of a local daemon which listens for connections by ----------------------------------------------------------------------------- 11.4. wmonload wmomload is a simple, but handy and small dockapp for overviewing the load of ----------------------------------------------------------------------------- cluster nodes in a small openMosix-based cluster. 11.5. openMosixWebView openMosixWebView - Produces web charts for monitoring an openMosix cluster. openMosixWebView is a PHP script for monitoring an openMosix cluster via the WEB. It uses openMosixview's openMosixCollector logs. Download now the last release openmosixwebview-0.2.12.tar.gz (16 Feb 2003) See openMosixWebView screenshots and running :-) Released under the GNU General Public License (GPL). See README and FAQ files. ----------------------------------------------------------------------------- Chapter 12. Common Problems 12.1. Introduction Although most of the issues in this chapter could be a part of the FAQ. There where the FAQ will give a short "how to solve" answer, we have taken a closer This is a replaced line ----------------------------------------------------------------------------- 12.2. My processes won't migrate This is an inserted line Help process XYZ doesn't migrate. Moshe Bar explains below why some processes migrate and why some don't. But before that you can always look in /proc/$pid This is an inserted line can't migrate. /, there often is a cantmove file which will tell you why a certain process Processes can also be locked. You can check if a process is locked with: cat /proc/$PID/lock where $PID is the processid of the process in question. Now listen to what Moshe himself has to say about this topic. Often people have the same kernel but on a different distribution, say a mixed environment of RedHat and Debian ,rc scripts from different distros tend to start openmosix differently. Some implementations completely modify / etc/inittab to start all daemons (and their children) with This is a replaced line . So that they won't migrate. Therefore all these processes have a 1 in /proc /pid/lock when you start. You can force them to migrate by writing a 0 to this file . Ok, this simple program should always migrate if launched more times than number of local CPUs. So for a 2-way SMP system, starting this program 3 times will start migration if the other nodes in the cluster have at least int main() { the same speed like the local ones: unsigned int i; while (1) { i++; This is an inserted line } return 0; } On a Pentium 800Mhz CPU it takes quite a while to overflow. This sample program with content like this will never migrate: This is a replaced line #include #include ... key_t key; /* key to be passed to shmget() */ int shmflg; /* shmflg to be passed to shmget() */ int shmid; /* return value from shmget() */ int size; /* size to be passed to shmget() */ ... key = ... size = ... shmflg) = ... if ((shmid = shmget (key, size, shmflg)) == -1) { perror("shmget: shmget failed"); exit(1); } else { (void) fprintf(stderr, "shmget: shmget returned %d\n", shmid); exit(0); } ... Program using pipes like this do migrate nicely: int pdes[2]; This is an inserted line pipe(pdes); if ( fork() == 0 ) { /* child */ read( pdes[0]); /* read from parent */ close(pdes[1]); /* not required */ ..... } else { close(pdes[0]); /* not required */ write( pdes[1]); /* write to child */ ..... MODIFIED Programs using pthreads since version 2.4.17 don't migrate, however } they don't segfault anymore. // Very simple program demonstrating the use of threads. // // Command-line argument is P (number of threads). // // Each thread writes "hello" message to standard output, with // no attempt to synchronize. Output will likely be garbled. // #include #include // has exit(), etc. #include // has usleep() #include // has pthread_ routines // declaration for function to be executed by each thread This is an inserted line This is an inserted line // ---- Main program ------------------------------------------------- int main(int argc, char* argv[]) { This is an inserted line cerr << "Usage: " << argv[0] << " numThreads\n"; exit(EXIT_FAILURE); } int P = atoi(argv[1]); This is a replaced line // since they're shared among threads). int * threadIDs = new int[P]; threadIDs[i] = i; // Start P new threads, each with different ID. pthread_t * threads = new pthread_t[P]; pthread_create(&threads[i], NULL, printHello, (void *) &threadIDs[i]); for (int i = 0; i < P; ++i) // Wait for all threads to complete. for (int i = 0; i < P; ++i) pthread_join(threads[i], NULL); // Clean up and exit. delete [] threadIDs; delete [] threads; cout << "That's all, folks!\n"; return EXIT_SUCCESS; } This is an inserted line This is a replaced line // ---- Code to be executed by each thread --------------------------- This is an inserted line // post: "hello" message printed to standard output. This is a replaced line This is a replaced line // pointless pause, included to make the effects of cout << "hello, world, "; // synchronization (or lack thereof) more obvious This is a replaced line This is an inserted line cout << "from thread " << *myID << endl; This is a replaced line } (sockets are not migrated with the process however, files are migrated if Programs using all kinds of file descriptors, including sockets do migrate using oMFS/DFSA) This is a replaced line Inc.) (all above code is by Moshe as Moshe Bar or by Moshe as CTO of Qlusters, Please also refer to the man pages of openMosix , they also give an adequate explanation why processes don't migrate. try to allow locked processes to migrate by simply putting If for some reason your processes stay locked while they shouldn't. You can # tell shells to allow subprocs to migrate to other nodes echo 0 > /proc/self/lock into "/etc/profile" Warning : this fix will allow all process to migrate not just the ones you want. To only allow specific process to migrate use 'mosrun This is a replaced line ----------------------------------------------------------------------------- 12.3. I don't see all my nodes 'same-kernel' refers to the version. You can build different kernel images of First of all , are you using the same kernel version on each machine ? The the same source version to meet the hardware/software needs of a given node. cluster, all your machines should have the openmosix-x.x.x-y kernel installed, in contrast to having one machine running openmosix-x.x.z-x, another running openmosix-x.x.x-y, another running openmosix x.x.x-z, and so on and so forth warn you that mosix is not running? When you run mosmon, press t to see the total of machines running. Does it If yes, then make sure your machine's ip is included in /etc/mosix.map (don't use 127.0.0.1 - if your machine's ip is such, then you probably have problems with your dhcp server/nameserver). If it does not tell you that mosix is not running, see what machines show up. Do you see only your machine? This is an inserted line If yes, then your machine is most likely running a firewall and is not letting openmosix through. If not, then the problem is most likely with the machine that doesn't show up. Also: Do you have two nic cards on a node? then you have to edit the /etc non-cluster_ip cluster-hostname.cluster-domain cluster-hostname You might also need to set up a routing table, which is a whole different subject. Maybe you used different kernel-parameters on each machine? Especially if you use the 'Support clusters with a complex network topology' option you should take care that you use the same value for the also appearing option 'Maximum network-topology complexity support' on each machine. ----------------------------------------------------------------------------- 12.4. I often get errors: No such process I often get the error bash: child setpgid (4061 to 4061): No such process what does this mean ? The above line meas that the shell you were using has acutallly migrated to another node ? This printout from bash is caused by a bug in old version of openmosix, but a fix has been commited. (Muli Ben-Yehuda mulix@actcom.co.il) ----------------------------------------------------------------------------- 12.5. DFSA ? MFS ? People often get confused about what exactly MFS and DFSA are. As discussed before in the howto MFS is the feature of openMosix that enables you access This is a replaced line mostly mounted on /mfs . A common misunderstanding is that you need MFS in order to have openMosix working, this is not true, however it can make things easier. With DFSA enabled, system calls will be executed on the remote node withouth migrating the process back to it's home node. This behaviour (direct way around (which is common). If DFSA is not enabled MfS is "just" a filesystem access) causes processes migratiing to the data and not the other non-caching network-filesystem. Very generally speaking, if you don't have DFSA turned on, each and every I This is an inserted line happens to be residing on the node where the process finds itself then the I/ O will happen locally. A very common error is that people mix kernels with DFSA enabled and This is a replaced line enabled. This information can be obtained by typing cat /proc/hpc/admin/version This is a replaced line 12.6. Python Troubles This is an inserted line Some people have reported problems with Python, closer research showed that these problems were not with openMosix but rather with glibc issues, however it seemed that issues manifested themselves especially in openMosix. This is a replaced line One user solved the problem by removig /lib/i686/lib* on his machine and let the applications link dynamically against /lib/libpthread (and other) However bugfixes in newer glibc versions combined with more recent openMosix version ----------------------------------------------------------------------------- seem to have solved these problems. Chapter 13. Hints and Tips 13.1. Locked Processes If for some reason you find your processes are always locked in your home node and you can't find the reason, you can put the following lines into your ~/.profile as a stop-gap measure to automatically enable migration: if [ -x /proc/$$/lock ]; then echo 0 > /proc/$$/lock However, you should make an effort to find out what the problems is ----------------------------------------------------------------------------- 13.2. Choosing your processes You will probably want to test your setup before deciding which programs you machine and have a significantly faster machine has part of your Mosix want to enable migration for. For example, if you are running KDE2 on a slow cluster, you might find resource-hungry programs like kmail are migrated out. This is not a bad thing as such, however, it can lead to a brief moment when your writing is not displayed on the screen immediately. ----------------------------------------------------------------------------- 13.3. Java and openMosix separate process. Threads other than Java green thread JVM's cannot be migrated by Linux, so openMosix cannot migrate programs that use them. If you have the source so your Java application you might be able to compile the application native. In this case you might be able to migrate your applications to another node. amongst other topics with Java an dopenMosix. http://www.democritos.it/events Gian Paolo Ghilardi wrote a paper titled Consideration on OpenMosix it deals /openMosix/papers/crema.pdf ----------------------------------------------------------------------------- 13.4. openMosix and Hyperthreading Basically openMosix performance increases with the current Linux scheduler when Hypethreading is disabled. You can do this by either entering 'noht' as a boot option or disabling HT in the bios. For those who are still wondering what hypetrheading is : Intel explains it 13.5. openMosix and Firewalls People often have questions regarding openMosix and firewalls. Amit helped me out on this matter: from hpc/comm.c: #define INFO_DAEMON_PORT 0x3415 3412, 'cos of the network-host byte ordering conversions.. Read the IP/TCP/ converted to decimal, they are: 4660 and 5428 resp. (convert 1234, and not UDP RFCs for info) This is an inserted line This is an inserted line the mig_daemon port is a tcp port, the info_daemon port is udp. Hence tcp/ 4660 and udp/5428, Matt also mentions tcp/723 somewhere. ----------------------------------------------------------------------------- Chapter 14. (stress)Testing your openMosix installation 14.1. A small Test Script The fastest way to test your openMosix cluster is by creating a small script with the following content. awk 'BEGIN {for(i=0;i<10000;i++)for(j=0;j<10000;j++);}' & This is an inserted line This is a replaced line start op mosmon and I launch this script for a zillion times as in for i in `ls /etc/` ; do ./test_mosix ; done This is a replaced line [howto6] Just pkill awk on the home node to kill'em all ;) ----------------------------------------------------------------------------- 14.2. Perl Proggie by Charles Nadeau Perl program to test an openMosix Cluster. Here is a is quick program I wrote to test an openMosix cluster. This is taken from a posting I made to the openMosix-devel mailing list on March 6th, 2002: "Charles wrote this little program (in Perl) to stress test his home cluster (3 P200MMX and a P166). It is a program simulating different sets of This is an inserted line stocks in a portfolio for a given period of time. The code is well documented and it should be easy to add/remove stocks and change the average monthly yield and standard deviation for each stock. Since the problem of portfolio optimization cannot be solved analytically, it simulate a lot of portfolios and report the results at the end. Please note that this program does not start. I plan to add more code at the end of the program to improve the This is a replaced line This is a replaced line simulation part works very well. In order to take advantage of the parallelism offered by openMosix, it uses the Perl module Parallel::? the machines of the cluster (it also require another module for the statistical calculations, don't forget to install both, I provide the URLs in This is a replaced line Cheers!" # this mill unlock this process and all tis childs sub unlock { open (OUTFILE,">/proc/self/lock") || die "Could not unlock myself!\n"; print OUTFILE "0"; } unlock; sub cpucount { $CLUSTERDIR="/proc/hpc/nodes/"; # this will count the nodes opendir($nodes, $CLUSTERDIR); $howmany=0; while(readdir($nodes)) { $howmany++; } $howmany--; $howmany--; closedir ($nodes); return $howmany; } my $processes=cpucount; $processes=$processes*3; This is a replaced line #Portfolio.pl, version 0.1 #Perl program that simulate a portfolios for various stock composition for a given period of time #We run various scenarios to find the mix of assets that give the best performance/risk ratio #Can be used to test an OpenMosix cluster #This method is base on the book "The intelligent asset allocator" by William Bernstein #This program is licensed under GPL #Author: Charles-E. Nadeau Ph.D., (c) 2002 #E-mail address: charlesnadeau AT hotmail DOT com use Parallel::ForkManager; #We use a module to parallelize the calculation #Available at http://theoryx5.uwinnipeg.ca/mod_perl/cpan-search?filetype=%20distribution%20name%20or%20description;join=and;arrange=file;download=auto;stem=no;case=clike;site=ftp.funet.fi;age=;distinfo=2589 use Statistics::Descriptive::Discrete; #A module providing statistical values #Available at http://theoryx5.uwinnipeg.ca/mod_perl/cpan-search?new=Search;filetype=%20distribution%20name%20or%20description;join=and;arrange=file;download=auto;stem=no;case=clike;site=ftp.funet.fi;age=;distinfo=2988 srand; #We initialize the random number generator #Initializing constant This is a replaced line $NumberOfMonth=100000; #Number of month for wich to run the simulation $NumberOfStock=6; #Number of different stocks in the simulation #TODO: Read the stock details from a file #Portfolio to simulate $Stock[0][0]="BRKB"; #Stock ticker $Stock[0][1]=0.01469184; #Stock average monthly return This is an inserted line $Stock[0][2]=0.071724934; #Stock average monthly standard deviation $Stock[1][0]="TEST "; #Stock ticker $Stock[1][1]=-0.01519; #Stock average monthly return $Stock[1][2]=0.063773903; #Stock average monthly standard deviation $Stock[2][0]="SPDR"; #Stock ticker $Stock[2][1]=0.008922718; #Stock average monthly return $Stock[2][2]=0.041688404; #Stock average monthly standard deviation $Stock[3][0]="BRKB"; #Stock ticker $Stock[3][1]=0.01469184; #Stock average monthly return $Stock[3][2]=0.071724934; #Stock average monthly standard deviation $Stock[4][0]="TEST "; #Stock ticker $Stock[4][1]=-0.01519; #Stock average monthly return $Stock[5][1]=0.008922718; #Stock average monthly return $Stock[5][0]="SPDR"; #Stock ticker $Stock[5][2]=0.041688404; #Stock average monthly standard deviation my $pm = new Parallel::ForkManager($NumberOfSimulation); #Specify the number of threads to span $pm->run_on_start( print "started, pid: $pid\n"; sub { my ($pid,$ident)=@_; } ); #We initialize the array that will contain the results @Results=(); This is a replaced line for $j (0..$NumberOfStock+3){ $Results[$i][$j]=0.0; #Equal to 0.0 to start } } for $i (0..$NumberOfSimulation-1){ #Loop on the number of simulation to run $pm->start and next; #Start the thread $Results[$i][0]=$i; #The first column of each line is the number of the simulation This is a replaced line #Here we calculate the portion of each investment in the portfolio for a given simulation for $j (0..$NumberOfStock-2){ #We loop on all stock until the second to last one #TODO: Replace rand by something from Math::TrulyRandom $Ratio[$j]=rand($TotalRatio); This is a replaced line This is an inserted line This is a replaced line $Ratio[$NumberOfStock-1]=$TotalRatio; #In order to have a total of the ratios equal to one, we set the last ratio to be the remainder } $Results[$i][$NumberOfStock]=$Ratio[$NumberOfStock-1]; #We store the ratio associated to this stock. Special case for the last stock my $stats=new Statistics::Descriptive::Discrete; #We initialize the module used to calculate the means and standard deviations $InvestmentValue=1; #Initially the investment value is 1 time the initial capital amount. for $j (1..$NumberOfMonth){ #Loop on the number of months This is an inserted line This is an inserted line $MonthlyGrowth[$j]=0.0; #By how much did we grow this month. Initially, no growth yet. for $k (0..$NumberOfStock-1){ $MonthlyGrowth[$j]=$MonthlyGrowth[$j]+($Ratio[$k]*((gaussian_rand()*$Stock[$k][2])+$Stock[$k][1])); #We had the growth for this stock to the stock already calculated for the preceding stocks } $stats->add_data($MonthlyGrowth[$j]); #Add the yield for this month so we can later on have the mean and standard deviation for this simulation } $Results[$i][$NumberOfStock+1]=$stats->mean(); #Calculate the average monthly growth $Results[$i][$NumberOfStock+2]=$stats->standard_deviation(); #Calculate the standard deviation of the monthly growth $pm->finish; #Finish the thread $pm->wait_all_children; #We wait until all threads are finished } #Printing the results print "Simulation "; print "Ratio$Stock[$j][0] "; } print " Mean StdDev YieldRatio\n"; for $i (0..$NumberOfSimulation-1){ printf "%10d ", $Results[$i][0]; for $j (1..$NumberOfStock){ This is a replaced line } This is an inserted line if($Results[$i][$NumberOfStock+2]!=0) { printf "%5.4f %5.4f %5.4f\n", $Results[$i][$NumberOfStock+1], $Results[$i][$NumberOfStock+2], ($Results[$i][$NumberOfStock+1]/$Results[$i][$NumberOfStock+2]); } else { printf "%5.4f %5.4f %5.4f\n", $Results[$i][$NumberOfStock+1], $Results[$i][$NumberOfStock+2], 0; } } #Subroutines #Subroutine to generate two numbers normally distributed #From "The Perl Cookbook", recipe 2.10 sub gaussian_rand { my ($u1, $u2); my ($g1, $g2); This is an inserted line do { $u1=2*rand()-1; $u2=2*rand()-1; $w=$u1*$u1+$u2*$u2; } while ($w>=1 || $w==0); #There was an error in the recipe, I corrected it here $w=sqrt(-2*log($w)/$w); $g1=$u2*$w; return wantarray ? ($g1,$g2) : $g1; ----------------------------------------------------------------------------- 14.3. the openMosix stress-test This is a replaced line This is a replaced line 14.3.1. General description This stress test is made to test an openMosix cluster + kernel. It will perform several application + kernel tests for checking the stability and other features of openMosix (e.g. process migration, mfs, ...). During this applications before starting it. When it finished it generates a fully test the cluster will be mostly loaded so you should stop other running detailed report about each component which was tested. ----------------------------------------------------------------------------- 14.3.2. Detailed description This is a replaced line of the whole system. In the following part you will find a description of each test-application: 1024 bits of key length. It is destributed into as many processes as   * distkeygen: This applicaton is used to generate 4000 RSA key pairs with processors in your openMosix cluster via fork. Requires : gcc compiler and OpenSSL library Copyright (C) 2001 Ying-Hung Chen (GPL) http://www.yingternet.com/mosix   * portfolio 'portfolio' is a perl program that simulate a portfolios for various stock composition for a given period of time. This method is base This program is licensed under GPL Author: Charles-E. Nadeau Ph.D., (c) This is a replaced line   * eatmem : Simply calculates sin+sqrt from a value 1000000 times and tests is started as many times at once as many processors you have in   * forkit: The 'forkit' test is similar to the 'eatmem' test but uses fork to create multiple process (3*[processors_in_your_openMosix_cluster]) expect it does not write to files.   * mfstest This will create a 10MB file and copy it to all nodes back and This is an inserted line forth. It is for checking the oMFS capabilities.   * kernel syscall test: The Linux Test Project is a joint project with SGI, IBM, OSDL, and Bull with a goal to deliver test suites to the open source Linux. The Linux Test Project is a collection of tools for testing the kernel by bring test automation to the kernel testing effort. Interested Linux kernel and related features. The goal is to improve the Linux open source contributors are encouraged to join the project. For more informations visit : http://ltp.sf.net   * moving: The 'moving.sh' will move the 'start_openMosix_test.sh' around each node in your openMosix cluster while running the stress-test itself. So 'start_openMosix_test.sh' will migrate every minute to another node during the test-run. Dependent on how long the test will run on your cluster it will be migrated 20-40 times. ----------------------------------------------------------------------------- 14.3.3. Installing the strestest suite First of all download the rpm or source package from http:// www.openmosixview.com/omtest/   * using the source package : Unzip and untar the openMosix stress-test with the following commands in e.g. /usr/local: tar -xvf omtest.gz gunzip omtest.tar.gz Then 'cd' into the /usr/local/omtest directory and execute: ./compile_tests.sh This will install the required perl modules and compile the test-programs. The installation of these modules requires root-privileges. Later you can run the openMosix stress-test also as a regular user. (you maybe have to delete old temporary files from root's test-runs in /tmp because you won't have the permission to overwrite it as regular user) You are ready to run the test now with the command: ./start_openMosix_test.sh   * using the RPM-package There are some requirements to be met when installing the omtest.rpm, you will need e.g expect and compat-libstdc++-7.3-2.96.110 (If you are on RH rpm -ihv omtest.rpm Now you can start the openMosix stress test with the command: start_openMosix_test.sh the RPM wil allso install some perl modules). (The RPM-package will be installed in /usr/local/omtest) (Please not that ----------------------------------------------------------------------------- 14.3.4. Running the tests [root@dhcp51 omtest]# ./start_openMosix_test.sh starting the openMosix stress test now! This is an inserted line the results will be saved in : /tmp/openMosix-stress-test-report-03/16/2003-11:27:02.txt Please mount oMFS before running this openMosix-test oMFS is not mounted at /mfs! oMFS-test will be disabled. You will find instructions how to configure and use oMFS at: http://howto.ipng.be/openMosix-HOWTO/x222.htm#AEN243 This is a replaced line IV. Running Applications on openMosix Table of Contents 15.1. Introduction 15. Improving Compiling Performance This is an inserted line 16. Imaging with openMosix 16.1. Introduction 16.2. Povray 17. BioInformatics and openMosix 17.1. Introduction 17.2. Blast ----------------------------------------------------------------------------- Chapter 15. Improving Compiling Performance 15.1. Introduction This is a replaced line This is an inserted line Lots of people try to use openMosix as a kind of compile farm, ofthen they come back very disappointed. This chapter of the howto will try to explain in which cases your compilations will benefit from openMosix and how to improve your successlevel. First of all you have to remember 1 thing. openMosix will not migrate all migration to another node. For compiling this means that a process has to last long enough. Kernel compiles typically consist of numerous short This is an inserted line compiles, each of them not being long enough to acutally migrate. ----------------------------------------------------------------------------- 16.1. Introduction This is a replaced line This Section is a Work in Progress Computer Graphics have always been applications that required a lot of CPU power, this hasn't changed. Within this chapter I wil demonstrate with some ----------------------------------------------------------------------------- 16.2. Povray The Persistence of Vision Raytracer is a high-quality, totally free tool for creating stunning three-dimensional graphics. simulating the way rays of light travel in the real world. However it does Ray-tracing is a rendering technique that calculates an image of a scene by its job backwards. In the real world, rays of light are emitted from a light source and illuminate objects. The light reflects off of the objects or passes through transparent objects. This reflected light hits our eyes or perhaps a camera lens. Because the vast majority of rays never hit an observer, it would take forever to trace a scene. These kind of applications can be easily made parrallel by using pvmpovray. Pvmpovray expects to working on a Beowulf style cluster and spread it's load to other nodes using pvm. The openMosix way of doing this is the same, work fo you ! This is an inserted line however we just do this on 1 machine and have openMosix do the load spreading A GREAT Howto on PVM Povray will show you how to setup PVMPovray. Below is a small summary. $ cd pvmpov3_1g_2 $ tar xfz ../povuni_s.tgz $ tar xfz ../povuni_d.tgz This is a replaced line Trying to apply the patch. Searching for rejected files Now compile with aimk ( which is a wrapper script provided by the pvm rpm , buth which probably won't be in your path.(some of the readers might remember aimk from other platforms / applications ) If you are on a RH 8.0 box id moved libpng and zlib to .notused .. This in order to prevent version issues .. with other libpng and zlib versions. export PATH=$PATH:/usr/share/pvm3/lib I then run aimk newunix. Then we start pvm and quit it. The daemon stays export PVMROOt=/usr/share/pvm3 active.. And a last thing that is not known by a novice pvm user is that pvm does use with the complete pathname. its own paths, and you have to put pvmpov either in that path or launch it This is a replaced line +Oskyvase.tga +NT16 +NW64 +NH64 +v +w1024 +h768 Persistence of Vision(tm) Ray Tracer Version 3.1g.Linux.gcc Jakob Flierl - PVMPOV Version 3.1g.2 The POV-Ray Team(tm) is not responsible for supporting this version. This is an inserted line Copyright 1999 POV-Ray Team(tm) Never found section in file /usr/src/povray/pvmpov3_1g_2/povray31/include/. Initializing PVMPOV Spawning /usr/local/bin/pvmpov with 16 PVM tasks on 1 hosts... Waiting up to 120s for first slave to start... ...16 PVM tasks successfully spawned. This is an inserted line Slave 0 successfully started. Input file: skyvase.pov (compatible to version 3.1) Remove bounds........On Split unions........Off Parsing Options Library paths: /usr/local/lib/povray31 /usr/local/lib/povray31/include Output Options Image resolution 1024 by 768 (rows 1 to 768, columns 1 to 1024). This is a replaced line Graphic display.....Off Mosaic preview......Off Continued trace.....Off Allow interruption...On Pause when CPU usage histogram.Off done.....Off This is an inserted line Verbose messages.....On Tracing Options Bounding boxes.......On Bounding threshold: 25 This is an inserted line Quality: 9 Antialiasing........Off Light Buffer.........On Vista Buffer.........On Animation Options Clock value.... 0.000 (Animation off) PVM Options Block Width.... 64 Block Height... 64 PVM Tasks...... 16 This is an inserted line PVM Nice....... 5 PVM Arch....... PVM Slave...... /usr/local/bin/pvmpov PVM WorkingDir. /usr/src/povray/pvmpov3_1g_2/povray31 This is a replaced line All Streams to console..........On Debug Stream to console.........On Fatal Stream to console.........On Render Stream to console........On Statistics Stream to console....On Warning Stream to console.......On This is an inserted line Starting frame 0... Slave 1 at dhcp71.office.be.stone-it.com successfully started. Slave 2 at dhcp71.office.be.stone-it.com successfully started. Slave 3 at dhcp71.office.be.stone-it.com successfully started. Slave 4 at dhcp71.office.be.stone-it.com successfully started. Slave 5 at dhcp71.office.be.stone-it.com successfully started. Slave 7 at dhcp71.office.be.stone-it.com successfully started. Slave 6 at dhcp71.office.be.stone-it.com successfully started. Slave 8 at dhcp71.office.be.stone-it.com successfully started. Slave 9 at dhcp71.office.be.stone-it.com successfully started. Slave 11 at dhcp71.office.be.stone-it.com successfully started. Slave 12 at dhcp71.office.be.stone-it.com successfully started. Slave 13 at dhcp71.office.be.stone-it.com successfully started. This is a replaced line 0:00:53 86.46 of blocks complete.Not using dhcp71.office.be.stone-it.com Slave 15 at dhcp71.office.be.stone-it.com successfully started. for reassignment (77%) for reassignment (67%) Not using dhcp71.office.be.stone-it.com for reassignment (85%) 0:00:55 89.06 of blocks complete. 640 of 768 lines finished (in frame 0).Not using dhcp71.office.be.stone-it.com for reassignment (65%) 0:00:56 91.67 of blocks complete.Not using dhcp71.office.be.stone-it.com for reassignment (72%) 0:00:56 92.71 of blocks complete.Not using dhcp71.office.be.stone-it.com for reassignment (80%) 0:00:57 93.75 of blocks complete. Slave at dhcp71.office.be.stone-it.com has exited. 0:00:57 94.79 of blocks complete. Slave at dhcp71.office.be.stone-it.com has exited. Slave at dhcp71.office.be.stone-it.com has exited. This is an inserted line 0:00:58 95.83 of blocks complete. Slave at dhcp71.office.be.stone-it.com has exited. 0:00:58 96.35 of blocks complete. 672 of 768 lines finished (in frame 0).Not using dhcp71.office.be.stone-it.com for reassignment (77%) Slave at dhcp71.office.be.stone-it.com has exited. 0:00:58 97.14 of blocks complete. 688 of 768 lines finished (in frame 0). This is an inserted line Slave at dhcp71.office.be.stone-it.com has exited. 0:00:59 97.92 of blocks complete. Slave at dhcp71.office.be.stone-it.com has exited. 0:00:60 98.44 of blocks complete. 704 of 768 lines finished (in frame 0). 0:01:03 100.00 of blocks complete. 768 of 768 lines finished (in This is a replaced line frame 0). Finishing frame 0...rtw. 768 Waiting for remaining slave stats. This is a replaced line PVM Task Distribution Statistics: host name [ done ] [ late ] host name [ done ] [ late ] dhcp71.office.be.stone-it.com [ 5.21%] [ dhcp71.office.be.stone-it.com [ 8.85%] [ 0.00%]dhcp71.office.be.stone-it.com [ 7.81%] [ 0.07%] dhcp71.office.be.stone-it.com [ 8.85%] [ 1.17%]dhcp71.office.be.stone-it.com [ 4.69%] [ 0.00%] This is an inserted line 0.98%]dhcp71.office.be.stone-it.com [ 4.17%] [ 0.00%] dhcp71.office.be.stone-it.com [ 5.21%] [ 0.00%]dhcp71.office.be.stone-it.com [ 8.33%] [ 0.52%] dhcp71.office.be.stone-it.com [ 5.21%] [ 0.00%]dhcp71.office.be.stone-it.com [ 5.73%] [ 0.72%] dhcp71.office.be.stone-it.com [ 7.29%] [ This is an inserted line 2.73%]dhcp71.office.be.stone-it.com [ 4.17%] [ 0.00%] dhcp71.office.be.stone-it.com [ 5.21%] [ 0.00%]dhcp71.office.be.stone-it.com [ 6.77%] [ 0.13%] dhcp71.office.be.stone-it.com [ 4.69%] [ 0.00%]dhcp71.office.be.stone-it.com [ 7.81%] [ 0.00%] POV-Ray statistics for finished frames: skyvase.pov Statistics (Partial Image Rendered), Resolution 1024 x 768 Pixels: 303104 Samples: 303104 Smpls/Pxl: 1.00 ---------------------------------------------------------------------------- ---------------------------------------------------------------------------- Ray->Shape Intersection Tests Succeeded Percentage ---------------------------------------------------------------------------- Cone/Cylinder 1842227 900504 48.88 CSG Intersection 2742731 323346 11.79 CSG Union 1801008 521692 28.97 Plane 20223278 11233348 55.55 Quadric 1801008 1196533 66.44 Bounding Object 1842227 900504 48.88 Sphere 1801008 461786 25.64 This is a replaced line Calls to Noise: 1201944 Calls to DNoise: 2108954 Shadow Ray Tests: 2856188 Succeeded: 85620 Reflected Rays: 889606 Smallest Alloc: 9 bytes Largest: 20508 Peak memory used: 5643343 bytes ---------------------------------------------------------------------------- Time For Trace: 0 hours 1 minutes 7.0 seconds (67 seconds) Total Time: 0 hours 1 minutes 7.0 seconds (67 seconds) As you can see the application was splitted into differen parts and run This is a replaced line machines. I had good results with 2 to 3 times the the number of cpu's I had available ----------------------------------------------------------------------------- This is a replaced line Chapter 17. BioInformatics and openMosix 17.1. Introduction This is an inserted line ----------------------------------------------------------------------------- 17.2. Blast This is a replaced line This is a replaced line This is an inserted line the only alternative. First of all there are some known problems with this patch, and other versions of blast , blast tends to segfault sometimes, this mostly happens with the preformatted databases you download from the internet. If you run formatdb on a raw database these errors tend to go away. This is an inserted line that openMosix tends to speed up MPI, adding openMosix to this config might V. openMosix Development Table of Contents 18. Getting started with openMosix internals 18.1. Introduction This is an inserted line Chapter 18. Getting started with openMosix internals This is a replaced line this part has been written by Amit Shah There's not much documentation available right now for the kernel. I hope to write some in the coming weeks. Anyways, here's how the sources are laid out: The openMosix code resides largely in hpc/ and include/hpc. There are lots of directories to mm/, fs/, etc. You need to read up the code which interests patches to the core kernel files everywhere, right from the arch/i386 you and think that would matter for the present situation (that shouldn't be a problem, since you've done kernel coding). here's what you should expect in each of the source files:   *  hpc/badops.c: one file to handle all the bad operations: mostly return err codes   * hpc/balance.c: The load balancer code (load + mem usage + n/w usage) This is an inserted line   *  hpc/comm.c: The intra-cluster communication setup   *  hpc/config.c: The config code for openMosix: after you run the startup script   *  hpc/decay.c: decay (age) the stats and info collected from other nodes   *  hpc/deputy.c: Code executed on the deputy: service remote syscalls (ie. after the process has migrated), signals, etc. This is an inserted line   *  hpc/dfsa.c: Direct File System Access code: the distributed file system   *  hpc/div.c: the algorithms to do floating point divisions   *  hpc/freemem.c: to keep track of free, avl. memory and to free it if need be. hugely taken from the Linux mm/ code.   *  hpc/hpcadmin.c: tune openMosix admin values (through /proc/hpc)   *  hpc/hpcproc.c: The /proc/hpc code is handled here This is an inserted line   *  hpc/info.c: The info daemon: sends and receives (multicast) load+mem usage stats throughout the cluster This is a replaced line   *  hpc/kernel.c: most of the "core" code: all the important algorithms, decisions, etc. made here.   *  hpc/load.c: calculation of local load, etc.   *  hpc/mig.c: Code that handles the migration. Code in this file is invoked on any migration: deputy->remote, remote->deputy; remote->remote   *  hpc/prequest.c: handles the process's requests: signals, more memory, etc.   *  hpc/remote.c: Code executed when the process is on the remote: syscalls handling on remote, passing control to deputy, etc.   * hpc/service.c: setting up daemons, getting memory, etc.   * hpc/syscalls.c: handles all the remote syscalls here   * hpc/ucache.c: handles the ucache: mostly mm/, fs/ stuff. the other files like auto_syscalls.c, alternate.c are generated at compile time. This is a replaced line Table of Contents 19.1. General 19. the openMosix FAQ 19.2. Getting, building, installing and running openMosix 19.3. Kernel Questions 19.4. File Systems 19.6. Resources 19.7. Miscellaneous This is a replaced line 20. PlumpOS FAQ 20.1. Frequently Asked Questions This is a replaced line ----------------------------------------------------------------------------- Chapter 19. the openMosix FAQ 19.1. General 19.1.1. What is openMosix?: 19.1.2. What does the term single-image clustering mean?: 19.1.4. Are there any mailing lists for openMosix?: 19.1.5. Can I contribute to the openMosix project?: 19.1.6. Who is the copyright holder of openMosix?: 19.1.7. Is openMosix a fork of MOSIX?: 19.1.8. Why did openMosix split from the MOSIX group?: This is an inserted line 19.1.1. What is openMosix?: The openMosix system is a Linux kernel extension for single-image clustering. It extends the outstanding MOSIX project, but is instead licensed under the This is an inserted line GNU General Public License (GPL). This is a replaced line 19.1.2. What does the term single-image clustering mean?: There are many varieties of clusters, and a single-image cluster has multiple copies of a single operating system kernel. 19.1.3. Does openMosix have a homepage?: Yes. It is at www.openMosix.org. The SourceForge project page is at http:// openmosix.sf.net/ 19.1.4. Are there any mailing lists for openMosix?: This is a replaced line This is an inserted line 1. For general discussion, use mailto: openmosix-general@lists.sourceforge.net, whose general information page 2. For developers, use mailto:openmosix-devel@lists.sourceforge.net, whose listinfo/openmosix-devel This is an inserted line 3. Italian Language openMosix Mailing List hosted by Democritos (the INFM National Simulation Center in Trieste) This is a replaced line 19.1.5. Can I contribute to the openMosix project?: This is a replaced line Linux kernel maintenance system, Moshe Bar appoints official maintainers and then gives these maintainers the commit bit to the openMosix CVS source tree, This is a replaced line Right now we are looking for more experienced kernel hackers to work on new features like checkpoint/restart. Write to mailtomoshe@moelabs.com if you would like to become an openMosix developer. 19.1.6. Who is the copyright holder of openMosix?: All MOSIX code is copyright by Professor Amnon Barak of Hebrew University of openMosix system does not contain any non-GPL (i.e. MOSIX) code. Jerusalem. All openMosix code is copyright by Moshe Bar, Tel Aviv. The 19.1.7. Is openMosix a fork of MOSIX?: Originally, openMosix was a fork of MOSIX, but it has evolved into an advanced clustering platform. The openMosix system no longer contains any non-GPL (i.e. MOSIX) code. Compared to MOSIX, a number of features were added: A port to the UML (User-mode Linux) architecture New and cleaner migration code A better load balancer Much reduced kernel latencies A greatly simplified installation processes that uses RPM packaging A wealth of documentation This is an inserted line 19.1.8. Why did openMosix split from the MOSIX group?: This is an inserted line The principal issue was that MOSIX was not licensed with an Open Source license. ----------------------------------------------------------------------------- 19.2. Getting, building, installing and running openMosix This is a replaced line 19.2.1. Where do I get openMosix?: 19.2.2. Can I mix MOSIX and openMosix nodes in the same cluster?: 19.2.3. How do I build openMosix?: 19.2.4. What are userland tools?: 19.2.1. Where do I get openMosix?: The RPMs and source for openMosix are available from our Downloads/Files Section. Please read the release notes first! openMosix packages are available. Also, Gentoo Linux's emerge sys-apps/openmosix-user and Debian GNU/Linux 19.2.2. Can I mix MOSIX and openMosix nodes in the same cluster?: No. Just like the older MOSIX, you should not mix nodes because the protocols are subject to unannounced changes from version to version. In addition, every new version has bug fixes which warrant updating to the new kernels. This is a replaced line This is a replaced line 1. Start by unpacking both the Linux kernel sources and the corresponding openMosix distribution in a directory, say /usr/src. This is an inserted line $ cd /usr/src; tar xzf linux-2.x.xx.tar.gz ;gunzip openMosix2.x.xx.gz This is an inserted line This is a replaced line This is an inserted line 3. Apply the openMosix patches to the pristine Linux kernel sources with $ patch -p1 openMosix2.x.xx-x The directory /usr/src/linux-2.x.xx now contains the 2.x.xx kernel sources with the openMosix patches. Compile and install the resulting kernel as usual. 19.2.4. What are userland tools?: Userland tools are a collection of administrative tools used to examine and ----------------------------------------------------------------------------- This is a replaced line 19.3. Kernel Questions 19.3.1. What kernel versions does openMosix support?: 19.3.2. I'm trying to compile an openMosix-patched kernel. What compiler version should I use?: 19.3.3. I've compiled the kernel from the sources. How do I add it to the bootloader (LILO, GRUB, other)?: 19.3.4. I installed a Linux distribution and it says that its kernel is x.x.x-x. The openMosix README says not to mix kernel versions. Does that This is an inserted line 19.3.5. What does the phrase the same kernel on every machine mean? Does it mean that the openmosix-x.x.x-y RPM will not work on my machine?: This is an inserted line mean the same kernel version, or the same kernel image?: The latest Linux kernel supported is 2.4.19. Later versions of the 2.4 series This is a replaced line version should I use?: You should use gcc-2.95.3 as this is the recommended compiler for 2.4 kernels. This is a Linux kernel requirement, not just an openMosix This is an inserted line requirement. However, nothing precludes you from having, on the same system, gcc-2.95.3 for kernel compiles and gcc-3.x for non-kernel compiles. Additional notes: There are many kernel-related issues with gcc-3.x compilers. Inlining, optimization and page alignment do strange things to operating systems kernels. The standard Linux kernel is only guaranteed to compile and work properly with gcc 2.95.3. you should ensure you use gcc-2.96-74 or later. gcc-2.96-54 will not build optimization. Anything greater than -O2 may not be wise. Similarly, if you the kernel correctly. In addition, please pay attention to compiler choose to use gcc-2.95.x or derivatives, be sure not to use -fstrict-aliasing (which, depending on your version of gcc 2.95.x, may necessitate using -fno-strict-aliasing). 19.3.3. I've compiled the kernel from the sources. How do I add it to the bootloader (LILO, GRUB, other)?: Treat an openMosix kernel just like any other kernel. The openMosix system is simply an extension to the kernel, and will be treated like a standard kernel 19.3.4. I installed a Linux distribution and it says that its kernel is x.x.x-x. The openMosix README says not to mix kernel versions. Does that mean that the openmosix-x.x.x-y RPM will not work on my machine?: No. It means is that if you install openMosix on your cluster, all your openmosix-x.x.z-x, and openmosix-x.x.x-y, etc. This is a replaced line This is an inserted line mean the same kernel version, or the same kernel image?: the same source version to meet the hardware/software needs of a given node. It means the same kernel version. You can build different kernel images of ----------------------------------------------------------------------------- This is an inserted line 19.4. File Systems 19.4.2. Can somebody explain to me the difference bewteen MFS and DFSA, and 19.4.1. What's oMFS, how do I use, and where do I get it?: why I would need DFSA?: 19.4.1. What's oMFS, how do I use, and where do I get it?: The openMosix File System (oMFS) is the filesystem used by openMosix kernels. You get it by installing an openMosix kernel on the nodes of your cluster openMosix RPMs by default.) with oMFS enabled in the kernel-config. (It should be enabled in the You should also enable Direct Filesystem Access (DFSA) which allows a migrated process to execute many syscalls on the remote node locally without the need to migrate it back to its home node. oMFS features:   * Cache consistency This is an inserted line   * Timestamp consistency   * Link consistency The DFSA layer on top of oMFS makes sure to move the process to the data, instead of vice versa, whenever it makes sense. Please read more about oMFS and how to use it in earlier chapters of the HOWTO. This is an inserted line 19.4.2. Can somebody explain to me the difference bewteen MFS and DFSA, and why I would need DFSA?: DFSA stands for Direct File System Access and is an optimization. It allows remote proccesses to perform some file system system calls locally rather then sending them to their home node. MFS stands for Mosix File System and allows all nodes access to all node filesystems. DFSA runs on top of a cluster filesystem, in this case MFS. ----------------------------------------------------------------------------- 19.5. Programming openMosix 19.5.1. Generally, how do I write an openMosix-aware program?: This is an inserted line 19.5.1. Generally, how do I write an openMosix-aware program?: candidates for migration to another node. Write your programs as you normally would. Any processes that you spawn are 19.5.2. Can I write openMosix programs in perl?: Yes. Use the Parallel::ForkManager available from CPAN or directly from http: //www.cpan.org/authors/id/D/DL/DLUX/Parallel-ForkManager-x.x.x.tar.gz. ----------------------------------------------------------------------------- 19.6. Resources This is an inserted line 19.6.2. What other resources are available?: 19.6.1. Where can I find out technical details about openMosix?: 19.6.1. Where can I find out technical details about openMosix?: Here are some links:   * Brian Pontz's openMosix source code browser is at http:// This is a replaced line   * openMosix Internals: How openMosix Works is at http://sourceforge.net/ docman/display_doc.php?docid=10390&group_id=46729. This is an inserted line   * LTSP + openMosix: A Quick How-To is at http://sourceforge.net/docman/ display_doc.php?docid=10431&group_id=46729.   * Distributed OSs: General description of openMosix is at http:// sourceforge.net/docman/display_doc.php?docid=9562&group_id=46729. More links at the link section of the howto. 19.6.2. What other resources are available?: Here are some:   * The goal of the Parallel Execution Framework is to create a standalone device that allows easy clustering of standard PCs. See http:// parallel.sourceforge.net. ----------------------------------------------------------------------------- 19.7. Miscellaneous 19.7.2. Whats the difference between /etc/mosix.map , /etc/hpc.map , /etc/ 19.7.1. I don't see all my nodes. What's happening?: This is an inserted line openmosix.map: (127.0.0.1) is not there!: 19.7.4. I want to install openMosix but I am afraid my machines are too weak 19.7.3. setpe: the supplied table is well-formatted, but my IP address for this: 19.7.5. Under what conditions does VMWare work with openMosix: 19.7.6. What architectures besides x86 (e.g. SPARC, AXP, PPC...) are 19.7.7. Is there a parallel make tool for openMosix such as MPmake?: supported by openMosix?: 19.7.1. I don't see all my nodes. What's happening?: This is a replaced line When you run 'mosmon', press 't' to see the total number of machines running. Does it warn you that openMosix is not running? openmosix.map. Don't use 127.0.0.1. If you do, you will probably have If it does, then make sure your machine's IP address is included in /etc/ problems with your DHCP server or your DNS nameserver. If it does not, then see what machines show up. Do you see only your machine? This is an inserted line If yes, then your machine is most likely running a firewall and is not letting openMosix through. This is an inserted line If not, then the problem is most likely with the machine that doesn't show up. Also: Do you have two NIC cards on a node? If so, you have to edit /etc/ hosts to have a line that has the following format . You might also need to set up a routing table, which is an entirely different subject. Maybe you used different kernel-parameters on each machine? Especially if you use the 'Support clusters with a complex network topology' option you should take care that you use the same value for the also appearing option 'Maximum network-topology complexity support' on each machine. 19.7.2. Whats the difference between /etc/mosix.map , /etc/hpc.map , /etc/ openmosix.map: This is a replaced line This is a replaced line openMosix map name (and 'hpc' is still used for the /proc files in openMosix). The current map name is /etc/openmosix.map. This is a replaced line This is a replaced line You'll need to modify your /etc/hosts file. On Red Hat machines mostly the / 127.0.0.1 hostname.domain.com localhost etc/hosts file includes a line like This is a replaced line If hostname.domain.com has an IP address of 192.168.10.250, and if you looked up hostname.domain.com you might get 127.0.0.1 as an answer. This is a replaced line However, if you put 192.168.10.250 hostname.domain.com This is an inserted line 127.0.0.1 localhost in your /etc/hosts, openMosix won't complain. This is a replaced line for this: A machine is never too weak: I have three P200s (64MB each) and two P166s (one with 48MB and one with 192MB). Two of them are on 10Base-T and the other three on 100Base-T. Even with these antiquated machines and "heterogenous" network, I get perfect load balancing to run simulation programs that I write your machines are old. To us this is a nice feature of openMosix: you can add newer machines to an existing cluster as they become available. And you do in Perl. (Look at our ProgramToTestACluster"). Don't be held back by the fact not need to have all identical machines. That's fantastic! This is a replaced line 19.7.5. Under what conditions does VMWare work with openMosix: If you intend to run VMWare under openMosix so that openMosix would load-balance several instances of that (yes, that works). But, if you want to balance (that fails). run openMosix in several VMWare instances and let these instances load The first case works. The latter case does not work because VMware has a bug in its Pentium emulation that makes VMware crash (not openMosix, but the VMware binary) on the first migration. 19.7.6. What architectures besides x86 (e.g. SPARC, AXP, PPC...) are supported by openMosix?: Only IA-32 is currently supported. The port of openMosix to the Intel(r) Itanium(tm) IA-64 Processor Family is complete. Project plans for openMosix' second year include porting to the 64-bit AMD Opteron(tm) processor. 19.7.7. Is there a parallel make tool for openMosix such as MPmake?: many child proccesses to spawn. You can use regular gcc make. just use make -j #, where the # represents how ----------------------------------------------------------------------------- Chapter 20. PlumpOS FAQ 20.1. Frequently Asked Questions 20.1.1. What is PlumpOS? 20.1.3. How does it work? 20.1.4. Where can I go for more help? 20.1.5. Where is your homepage? 20.1.6. The penguins are gonna eat the poor halibut in the logo! 20.1.1. What is PlumpOS? PlumpOS is a mini linux distribution aimed at an easy way to add nodes to an openMosix cluster without a lot of work or even thought. The original idea came from (and current development is inspired by) ClumpOS, the now deceased project by Jean-David Marrow with a very similar goal for MOSIX clusters. This is a replaced line The first reason to use PlumpOS is to try out openMosix clusters in a simple and convenient fashion. The second reason is to add nodes to an existing This is an inserted line cluster without a lot of work. PlumpOS is in no way a replacement for enterprise clusters nor does it strive to be some kind of super duper cluster project; it merely presents a quick and easy way to turn your average 586+ PC into another node for your cluster. 20.1.3. How does it work? >From a developer's standpoint it's somewhat difficult but to the common user it into a computer's CDROM drive, boot up the PC (making sure there's a This is a replaced line This is a replaced line computer to distribute load/forked processes to. Because it's a child/slave node, you have to already have an openMosix box up and running to run tasks for PlumpOS computers to process. Without a master node, PlumpOS is pretty useless. That computer or some other network device must already be serving DHCP addresses or you'll find yourself configuring all your PlumpOS nodes by hand (which won't make for a happy network administrator - but then again that's why we have junior admins ^_^). Also, make sure you have the openMosix This is a replaced line autodiscovery daemon can be found into openMosix's userspace-tools package. 20.1.4. Where can I go for more help? Well, for PlumpOS specific help you should join the PlumpOS mailing list at http://lists.sourceforge.net/lists/listinfo/plumpos-general and ask a This is a replaced line This is an inserted line This is a replaced line PlumpOS. For openMosix related questions you should join the openMosix-general or openMosix-devel mailing list at http:// openmosix.sourceforge.net/mailing.html or visit irc.freenode.net #openMosix. 20.1.5. Where is your homepage? The official PlumpOS homepage is at http://PlumpOS.sourceforge.net/. You will find updated versions of this document there, as well as updated news, info, etc. 20.1.6. The penguins are gonna eat the poor halibut in the logo! This is an inserted line halibut was found deceased on an ice drift after having suffered a heart Yes, but not to worry: no halibuts were harmed in the making of the logo. The This is a replaced line system's mascot of choice is purely coincidental. ----------------------------------------------------------------------------- Appendix A. How to produce openMosix's Kernel RPM files A.1. How to produce openMosix's Kernel RPM files yourself" RPM packager , by Mirko Caserta This is an inserted line 1. Install a RedHat 8 (psyche) on your machine. This is so far the platform used to produce the rpms and it's known to do its job well 2. Get an updated copy of the "linux-openmosix" module from oM's CVS repository - details can be found at http://sourceforge.net/cvs/?group_id =46729 This is an inserted line 3. Get the tarball for the Linux Kernel sources which we are going to patch and put it in /usr/src/redhat/SOURCES - supposing we're talking about a 2.4.20 Kernel, get the linux-2.4.20.tar.bz2 file from one of the many http://www.kernel.org/ mirrors worldwide 4. Unpack the kernel tarball under /usr/src, ie: This is a replaced line # tar vxjf redhat/SOURCES/linux-2.4.20.tar.bz2 linux-openmosix module, for instance: This is an inserted line 5. Make a symbolic link to the directory where you've checked out the # ln -s /home/mcaserta/src/linux-openmosix/linux-openmosix \ /usr/src/linux-openmosix 6. copy the .spec file and all the .config files which are found in this This is an inserted line directory under /usr/src/redhat/SOURCES, ie: # cp /usr/src/linux-openmosix/configs/openmosix-kernel.spec \ /usr/src/redhat/SOURCES This is a replaced line # cp /usr/src/linux-openmosix/configs/*.config \ /usr/src/redhat/SOURCES 7. Now it's time to check the version numbers before we make the patch file: make sure the very first lines in /usr/src/linux-openmosix/Makefile and /usr/src/redhat/SOURCES/openmosix-kernel.spec have the correct kernel version and openMosix revision number 2.4.20 Linux Kernel and the 3rd release of openMosix): # cd /usr/src # diff -Naur --exclude=CVS --exclude=configs \ This is a replaced line # bzip2 /usr/src/redhat/SOURCES/openMosix-2.4.20-3 /usr/src/redhat/SOURCES/openMosix-2.4.20-3 9. At this point your /usr/src/redhat/SOURCES directory should look like: # ls /usr/src/redhat/SOURCES This is an inserted line This is an inserted line kernel-2.4.20-athlon-smp.config linux-2.4.20.tar.bz2 kernel-2.4.20-athlon.config kernel-2.4.20-i686-smp.config kernel-2.4.20-i386.config openMosix-2.4.20-3.bz2 kernel-2.4.20-i686.config openmosix-kernel.spec # cd /usr/src/redhat/SOURCES 10. Now you only need to rpmbuild the whole thing - what I usually do is: # rpmbuild -ba --target i386 openmosix-kernel.spec # rpmbuild -bb --target i686 openmosix-kernel.spec # rpmbuild -bb --target athlon openmosix-kernel.spec This is an inserted line This is an inserted line but you can easily build all rpms by calling: # rpmbuild -ba --target all_x86 openmosix-kernel.spec a) RPMS/i386/openmosix-kernel-2.4.20-openmosix3.i386.rpm This is a replaced line b) RPMS/i686/openmosix-kernel-2.4.20-openmosix3.i686.rpm c) RPMS/i686/openmosix-kernel-smp-2.4.20-openmosix3.i686.rpm d) RPMS/athlon/openmosix-kernel-2.4.20-openmosix3.athlon.rpm e) RPMS/athlon/openmosix-kernel-smp-2.4.20-openmosix3.athlon.rpm f) RPMS/i386/openmosix-kernel-source-2.4.20-openmosix3.i386.rpm h) SOURCES/openMosix-2.4.20-3.gz g) SRPMS/openmosix-kernel-2.4.20-openmosix3.src.rpm where: a) binary kernel package for i386 UP* machines b) binary kernel package for i686 UP* machines c) binary kernel package for i686 SMP** machines d) binary kernel package for athlon UP* machines e) binary kernel package for athlon SMP** machines f) source kernel package for any x86 machine (basically this is useful if you need to have the openMosix kernel headers) h) kernel patch file compressed with gzip g) source kernel package (see point 11) 12. The magic spell to obtain all the files back from the .src.rpm file is: # This is a replaced line Special thanks to Martin Høy for the help while I was trying to get the whole thing together. I hope you find this document useful. At least it is for me since I tend to forget things a few minutes after I've accomplished them :) This is a replaced line * UP = UniProcessor (i.e. one CPU) ** SMP = Symmetric Multi Processing (i.e. more than one CPU) ----------------------------------------------------------------------------- Appendix B. More Info This is an inserted line B.1. irc Some of the openMosix enthousiasts spend time online helping out people on irc. We are on irc.freenode.net on #openMosix. Just join us there to dicuss ----------------------------------------------------------------------------- B.2. Further Reading This is a replaced line ----------------------------------------------------------------------------- B.3. Translations Some people have been working on partial translations of this HOWTO, or just plain openMosix documentation in their own language. If you are working on a translation of this document let us know. ----------------------------------------------------------------------------- B.3.1. Chinese software.ccidnet.com/pub/disp/Article?columnID=732&articleID=25795&pageNO=1 This is a replaced line ----------------------------------------------------------------------------- B.3.2. Spanish translation of the HOWTO http://w3.akamc2.net/ ----------------------------------------------------------------------------- This is an inserted line Together with some collegues Miquel Catalán Coïthas been working on a spanish B.3.3. Russian Dmitry Katusubo translated the openmosix website and together with Yuri Prushinsky he also translated the openMosix HOWTO http://www.openmosix.org.ru ----------------------------------------------------------------------------- /docs/openMosix-HOWTO-multi/ B.4. Links   * openMosix on Sourcefourge This is a replaced line   * the openMosix FAQ   * the openMosix WIKI   * the openMosix Community Contributions   * openmosixview   *  Mosix Debian HOWTO This is a replaced line   *  K12 LTSP Mosix HOWTO This is a replaced line   * Mosix Mandrake Linux Terminal Server Project   *  openMOSIXVIEW, a GUI for managing openMosix-Cluster   *  User Mode openMosix, a virtual openMosix cluster running in User-mode   *  RxLinux, Web Interface for central configuration and management   *  LTSP+OpenMosix: A Mini How-To   * Charting the Land of Elliptic Curves using openMosix   * openMosixWebView   * Italian experiences with openMosix clusters Links to several papers This is a replaced line seen on the openMosix mailing list:   + Osservatorio Astronomico Cagliari (pdf)Securing an openMosix cluster in an untrusted networking environment.   + D.T.I. dell'Università di Milano, Polo didattico e ricerca, Crema (pdf) Experiences with Java and C programs on openMosix.   + Riello Group (pdf) Use of openMosix for parallel I/O balancing on storage (backup)   + INFM & Dept. of Physics University of Napoli (pdf) Documents a number This is a replaced line to software and projects.)   + Conecta srl (pdf) Contains more information on Kraken, the openMosix based game server. Included is how they monitor the cluster's temperature via lmsenors and how they improved internode This is a replaced line   * openMosix with Diskless client   * Use networked Linux systems to solve your computing challenges by Daniel Robbins ----------------------------------------------------------------------------- B.5. Mailing List   *  openMosix mailing list   * ClumpOS mailing list This is an inserted line ----------------------------------------------------------------------------- Appendix C. Credits The list of people who deserve credits for this HOWTO is long, I actually lost track of all the people that should be in here. I often add their names right next to the parts they have contributed. If you feel that your name is missing here do not hesitate to contact me and I'll gladly put your name into the list. I have to thank Scot W. Stevenson for all the work he did on this HOWTO This is an inserted line before I took over. He made a great start for this document. Assaf Spanier worked together with Scott in drafting the layout and the chapters of this HOWTO. and now promised to help me out with this document. Matthias Rechenburg Matthias Rechenburg should be thanked for the work he did on openMosixview and the accompanying documentation , which we included in this HOWTO. This is an inserted line Jean-David Marrow is the author of Clump/OS, he contributed the documentation on his distribution to the HOWTO. Bruce Knox is the maintainer of the openMosix website, he helps where he can and gives a This is an inserted line Evan Hisey This is a replaced line This is a replaced line Charles Nadeau for putting a lot of effort into putting extra documentation in the WIKI Louis Zechter Moshe Bar For writing the code he wrote and helping out with the docs wherever he knows the answers ! for getting started with the openMosix internals Mirko Caserta For sending in huge patches to this howto Ramon Pons for proofreading the howto and sending in some advice ----------------------------------------------------------------------------- Appendix D. GNU Free Documentation License Version 1.1, March 2000 Copyright (C) 2000 Free Software Foundation, Inc. 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. ----------------------------------------------------------------------------- 0. PREAMBLE The purpose of this License is to make a manual, textbook, or other written document "free" in the sense of freedom: to assure everyone the effective commercially or non-commercially. Secondarily, this License preserves for the freedom to copy and redistribute it, with or without modifying it, either author and publisher a way to get credit for their work, while not being This is an inserted line considered responsible for modifications made by others. This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the This is an inserted line GNU General Public License, which is a copyleft license designed for free This is an inserted line software. We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. textual work, regardless of subject matter or whether it is published as a This is a replaced line printed book. We recommend this License principally for works whose purpose is instruction or reference. ----------------------------------------------------------------------------- placed by the copyright holder saying it can be distributed under the terms This License applies to any manual or other work that contains a notice Any member of the public is a licensee, and is addressed as "you". or a portion of it, either copied verbatim, or with modifications and/or A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document's overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (For example, if the Document is in part a textbook of mathematics, be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them. The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. Front-Cover Texts or Back-Cover Texts, in the notice that says that the The "Cover Texts" are certain short passages of text that are listed, as Document is released under this License. represented in a format whose specification is available to the general public, whose contents can be viewed and edited directly and straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup is not Transparent. A copy that is not "Transparent" is called "Opaque". has been designed to thwart or discourage subsequent modification by readers Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a This is a replaced line human modification. Opaque formats include PostScript, PDF, proprietary formats that can be read and edited only by proprietary word processors, SGML and the machine-generated HTML produced by some word processors for output or XML for which the DTD and/or processing tools are not generally available, The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work's title, preceding the beginning of the body of the text. ----------------------------------------------------------------------------- 2. VERBATIM COPYING This is a replaced line or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the This is a replaced line may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3. You may also lend copies, under the same conditions stated above, and you may publicly display copies. ----------------------------------------------------------------------------- 3. COPYING IN QUANTITY If you publish printed copies of the Document numbering more than 100, and copies in covers that carry, clearly and legibly, all these Cover Texts: Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of This is a replaced line covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects. If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages. than 100, you must either include a machine-readable Transparent copy along If you publish or distribute Opaque copies of the Document numbering more publicly-accessible computer-network location containing a complete with each Opaque copy, or state in or with each Opaque copy a Transparent copy of the Document, free of added material, which the general public-standard network protocols. If you use the latter option, you must network-using public has access to download anonymously at no charge using take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public. Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document. ----------------------------------------------------------------------------- 4. MODIFICATIONS You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the Modified Version to whoever possesses a copy of it. In addition, you must do role of the Document, thus licensing distribution and modification of the these things in the Modified Version: This is an inserted line A. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You B. List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, together with at least five of the principal authors of the Document (all of its principal authors, if it has less than five). This is an inserted line C. State on the Title page the name of the publisher of the Modified Version, as the publisher. D. Preserve all the copyright notices of the Document. E. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices. F. Include, immediately after the copyright notices, a license notice giving the public permission to use the Modified Version under the terms of this License, in the form shown in the Addendum below. G. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the Document's license notice. I. Preserve the section entitled "History", and its title, and add to it an H. Include an unaltered copy of this License. item stating at least the title, year, new authors, and publisher of the Modified Version as given on the Title Page. If there is no section entitled "History" in the Document, create one stating the title, year, authors, and publisher of the Document as given on its Title Page, then add an item describing the Modified Version as stated in the previous sentence. access to a Transparent copy of the Document, and likewise the network J. Preserve the network location, if any, given in the Document for public locations given in the Document for previous versions it was based on. This is an inserted line This is a replaced line location for a work that was published at least four years before the gives permission. Document itself, or if the original publisher of the version it refers to K. In any section entitled "Acknowledgements" or "Dedications", preserve the section's title, and preserve in the section all the substance and tone of each of the contributor acknowledgements and/or dedications given therein. text and in their titles. Section numbers or the equivalent are not L. Preserve all the Invariant Sections of the Document, unaltered in their considered part of the section titles. M. Delete any section entitled "Endorsements". Such a section may not be included in the Modified Version. This is an inserted line N. Do not retitle any existing section as "Endorsements" or to conflict in title with any Invariant Section. If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version's license notice. These titles must be distinct from any You may add a section entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties--for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard. passage of up to 25 words as a Back-Cover Text, to the end of the list of You may add a passage of up to five words as a Front-Cover Text, and a Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any previously added by you or by arrangement made by the same entity you are one entity. If the Document already includes a cover text for the same cover, one, on explicit permission from the previous publisher that added the old one. The author(s) and publisher(s) of the Document do not by this License give This is an inserted line permission to use their names for publicity for or to assert or imply endorsement of any Modified Version. ----------------------------------------------------------------------------- 5. COMBINING DOCUMENTS You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice. The combined work need only contain one copy of this License, and multiple This is a replaced line multiple Invariant Sections with the same name but different contents, make This is an inserted line the title of each such section unique by adding at the end of it, in This is an inserted line parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section combined work. titles in the list of Invariant Sections in the license notice of the combine any sections entitled "Acknowledgements", and any sections entitled various original documents, forming one section entitled "History"; likewise "Dedications". You must delete all sections entitled "Endorsements." This is an inserted line ----------------------------------------------------------------------------- This is an inserted line This is a replaced line You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects. You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License regarding verbatim copying of that document. into the extracted document, and follow this License in all other respects ----------------------------------------------------------------------------- 7. AGGREGATION WITH INDEPENDENT WORKS A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or Document, provided no compilation copyright is claimed for the compilation. distribution medium, does not as a whole count as a Modified Version of the Such a compilation is called an "aggregate", and this License does not apply to the other self-contained works thus compiled with the Document, on account of their being thus compiled, if they are not themselves derivative works of the Document. If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one quarter of the entire aggregate, the Document's Cover Texts may be placed on covers that surround only the Document within the aggregate. Otherwise they must appear on covers around the whole aggregate. This is a replaced line 8. TRANSLATION This is a replaced line Translation is considered a kind of modification, so you may distribute Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License provided that you also include the original English version of this License. In case of a disagreement between the translation and the original English version of this License, the original English version will prevail. This is a replaced line 9. TERMINATION You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. terminate your rights under this License. However, parties who have received This is an inserted line ----------------------------------------------------------------------------- 10. FUTURE REVISIONS OF THIS LICENSE The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See [http://www.gnu.org/copyleft/] http:// www.gnu.org/copyleft/. Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and Each version of the License is given a distinguishing version number. If the conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation. ----------------------------------------------------------------------------- How to use this License for your documents To use this License in a document you have written, include a copy of the License in the document and put the following copyright and license notices just after the title page: This is an inserted line This is an inserted line Copyright (c) YEAR YOUR NAME. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with the Invariant Sections being LIST THEIR TITLES, with the Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST. A copy of the license is included in the section entitled "GNU Free Documentation License". of saying which ones are invariant. If you have no Front-Cover Texts, write If you have no Invariant Sections, write "with no Invariant Sections" instead "no Front-Cover Texts" instead of "Front-Cover Texts being LIST"; likewise for Back-Cover Texts. If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software This is a replaced line ----------------------------------------------------------------------------- Index