================================================================================ Note 27.0 Using Messages with VAXELN No replies FURILO::GIORGETTI 292 lines 21-AUG-1985 23:04 -------------------------------------------------------------------------------- +---------------+ +-----------------+ | d i g i t a l | | uNOTE # 027 | +---------------+ +-----------------+ +----------------------------------------------------+-----------------+ | Title: USING MESSAGES WITH VAXELN | Date: 19-JUN-85 | +----------------------------------------------------+-----------------+ | Originator: Christopher DeMers | Page 1 of 5 | +----------------------------------------------------+-----------------+ This Micronote discusses how VAXELN uses messages for inter-job communication. Included is an overview of the message architecture, benchmark data from MicroVAX II configurations and a sample program illustrating the message handling technique. In a VAXELN application, every job(Pascal,C or Macro-32 program) has a unique and protected virtual address space. Within a single processor, the kernel separates each job's virtual address space using the VAX memory management hardware. Within a network, each job's virtual address space is separated by virtue of the fact that the jobs may exist in the memories of different computer systems. One of the principal reasons for dividing an application into separate jobs is to aid the migration and distribution of the jobs within the network. In order for the jobs to work together on the same application, they must be able to share data, but the only way of moving data from the memory of one processor to another in a network is by packaging the data in a message and directing the network hardware to move the message to the destination memory. To make the network movement of data between jobs the same as in the non-network case, message passing is the principal means of inter-job communication. The kernel provides a number of facilities to make an efficient and transparent means of communication. The VAXELN MESSAGE object describes a block of memory that can be moved from one job's virtual address space to another's. The block of memory is called 'message data' and is allocated dynamically by the kernel from physically contiguous, page-aligned blocks of memory. A MESSAGE object and its associated message data are both created by calling the CREATE_MESSAGE kernel service. Message data is mapped into a job's P0 virtual address space, so it is potentially accessible to all the processes in the job. If a message is sent to a job on the sending job's local node, the kernel unmaps the 211 uNOTE # 027 Page 2 of 5 message data from the sending job's virtual address space and remaps it into the receiver's space. Instead of moving the data, Page Table Entries are remapped, resulting in very fast message transmission time. If a message is sent to a remote node, the kernel again unmaps the message data, but it remaps it into the appropriate network device driver job to send the message to the remote system. The reverse operations then cause the message data to be remapped in the receiver's space. The code necessary to send a message local to a machine or over the network is identical in both cases. A port name table parameter (NAME$LOCAL or NAME$UNIVERSAL) allows the programmer to define local or remote ports. The kernel maintains the local name list, while the name table for network-wide names is maintained by the name server. Names known throughout the network are guaranteed to be unique. Thus, jobs can initially be executed on one processor and later be segmented to run on multiple processors without modification of the program. Datagrams and Circuits ---------------------- Messages can be used two ways to transmit data: - One process can obtain the value of a port anywhere in the system (including other jobs) or in a different system running on a different Ethernet node. The process can send the port a message with the SEND datagram procedure. This is called the datagram method. - Any two ports (usually in different jobs) can be bound together to circuit form a circuit. In this case, having established the circuit, the sending process has one port of its own bound to another port, which usually is in a different job or on a different network node. The sender sends the message to its own port, and it is routed automatically to the other port in the circuit. Processes can both send to and receive from a circuit port. In the datagram method, as well as in the circuit method, a process can use the WAIT procedures to wait for the receipt of a message on the specified port. The datagram method requires no connection sequence as in circuits, but cannot guarantee that a message is actually received at the destination. However, it does guarantee that received messages are correct. At the cost of setting up and handling a circuit connection, circuits offer several advantages over the basic datagram method: Guaranteed delivery - Guaranteed delivery. The circuit method guarantees that either the message arrives at the destination port regardless of its location, or that the sender is notified that the message could not be delivered. 212 uNOTE # 027 Page 3 of 5 Flow control - Flow control. You can prevent a sending process from sending too many messages to a slower receiving process. If a port is connected in a circuit and is full, the sender is put into the waiting state until the port is no longer full (you can override this default). The implicit waiting performed by the SEND procedure evens the flow of messages between the transmitting process and the receiving process without having to explicitly program for the condition. Furthermore, circuits guarantee the sequence of message delivery as no other unconnected port can send a message to a connected port. Segmentation - Segmentation. Message can have any length (datagrams are limited to approx. 1500 bytes), and, if the transmission is across the network, the network services will divide the message into segments of the proper length, transmit the segments and reassemble them at the destination node. User interface - User interface via Pascal I/O routines. The OPEN procedure permits you to 'open' a circuit as if it were a file and to use the I/O routines such as READ and WRITE to transmit messages. There is no performance penalty with circuits for messages transmitted over the same network node and very little over the network. For full generality, programs should assume that the sending and receiving jobs may be distributed on different nodes in a network. Circuits are the preferred means of sending message in almost every practical case. Expedited Messages ------------------ An expedited message bypasses the normal flow-control mechanism and can be sent even if the receiving port already has its maximum number of messages. The message is received by the port before any normal data messages. The size of an expedited message is limited to a maximum of 16 bytes. Monitoring Network Integrity ---------------------------- Circuits can be used as a method for monitoring the status of a network connection. In addition to the circuit used to send data, another process could establish a secondary circuit with a process in a remote node. Neither process sends data, they only WAIT on their respective ports. When a node fails, the circuit is broken and the WAIT is satisfied. At this point exception handling routines could be invoked to handle this condition. This allows the programmer to separate the error handling code from the message transmission code. 213 uNOTE # 027 Page 4 of 5 Connecting with VAX/VMS nodes ----------------------------- The VAXELN procedure CONNECT_CIRCUIT can be used to request a connection with a VAX/VMS program on the same DECnet network. Instead of using a port as the destination for the connection, a VMS node and an object name (program name on the VMS node) is used. The program on the VMS node completes the connection by opening the "file" SYS$NET. A VAX/VMS node may also request a connection by specifying a node name and port name in the file portion of an OPEN statement. The VAXELN node complete the connection by issuing an ACCEPT_CIRCUIT statement. Performance Data ---------------- MicroVAX II, 5MB memory, connection via circuit All times in microseconds (us). Throughput in thousand bytes/second(KB/s). Delivery Message Size|Create|Delete| One Machine | Two Machines (bytes) | Time | Time |Time|Throughput| Time|Throughput ------------+------+------+----+----------+-----+---------- 0 | 131 | 144 |1094| N/A | 4479| N/A 512 | 361 | 255 |1611| 318 | 5271| 97 2048 | 450 | 274 |1885| 1087 | 8675| 236 8192 | 481 | 394 |2139| 3830 |27538| 298 214 uNOTE # 027 Page 5 of 5 PROGRAM msgsend(INPUT,OUTPUT); {++ { This program connects a circuit to the global name 'msgrecv', { sends a message to the receiver and waits for a reply. {--} VAR send_port,reply_port: PORT; msg,reply: MESSAGE; msg_data_ptr: ^VARYING_STRING(20); reply_data_ptr: ^VARYING_STRING(10); BEGIN CREATE_PORT(send_port); CONNECT_CIRCUIT(send_port, destination_name := 'msgrecv'); CREATE_MESSAGE(msg,msg_data_ptr); msg_data_ptr^ := 'message from msgsend'; SEND(msg,send_port); WAIT_ANY(send_port); RECEIVE(reply,reply_data_ptr,send_port); DELETE(reply); DISCONNECT_CIRCUIT(send_port); END; { program msgsend } PROGRAM msgrecv (INPUT,OUTPUT); {++ { This module runs a receive program that accepts a circuit,receives { a message and sends back a reply. {--} VAR reply_port: PORT; nam: NAME; msg,reply: MESSAGE; msg_data_ptr: ^VARYING_STRING(20); reply_data_ptr: ^VARYING_STRING(20); BEGIN CREATE_PORT(reply_port); CREATE_NAME(nam,'msgrecv',reply_port,table := name$universal); INITIALIZATION_DONE; ACCEPT_CIRCUIT(reply_port); WAIT_ANY(reply_port); RECEIVE(msg,msg_data_ptr,reply_port); DELETE(msg); CREATE_MESSAGE(reply,reply_data_ptr); reply_data_ptr^ := 'reply'; SEND(reply,reply_port); END; { program msgrecv } 215