Prototypes

The following prototypes demonstrate older designs than what is described on this main page, however what they demonstrate should still be doable albiet with slightly different probe names and arguments.

• Prototype #1, Oct 2006. TCP, UDP and IP providers. Also see the CEC2006 Demo.
• Prototype #2, Jun 2007. IP, IPv4, IPv6 providers.
• Prototype #3, Dec 2007. IP provider (covers IPv4 and IPv6).
Similar to prototype #2, but with the probe arguments as described in the arguments section.
• Prototype #4, Aug 2008. TCP provider.
Similar to the TCP probes from prototype #1. The source code is available.

Aims

The main aims of the network providers are:

• Support for common network protocols: TCP, UDP, IP.
• Everyday observability: connections/packets/bytes by address/port.
• Performance observability: throughputs, network latencies.
• Socket level tracing: associate events to PIDs.
• Further troubleshooting support: TCP flag analysis, header info.

Future enhancements and additions to the providers include:

• Other protocol support: ICMP, SCTP, ...
• Kernel code path latency probes.

News

The IP provider was integrated into Solaris Nevada build 93, and OpenSolaris. The official documentation is here: DTrace IP Provider.

The TCP provider is now being prepared for integration, and prototype documentation is available here: DTrace TCP Provider. Its usage is very similar to that which is documented here under Prototype #1.

Providers

The Network provider will be a collection of several DTrace providers, such as ipv4, ipv6 and tcp (there is no monolithic net provider). The proposed providers are,

provider	description
gld	This traces the generic LAN device layer, and shows link layer activity such as Ethernet frames. The probes allow frame by frame tracing.
arp	This traces ARP and RARP packets.
icmp	This traces ICMP packets, and provides the type and code from the ICMP header.
ip	This traces high level IP details for IPv4 and IPv6.
tcp	This traces the TCP layer, and shows what TCP activity is occuring.
udp	This traces UDP events.
sctp	This traces SCTP events.
socket	This traces the socket layer, close to the application. These probes fire in the same context as the corresponding process, and show data sent to or received from sockets.

Other providers will be added as required.

Probes

The following is the master plan for the net providers - what probes could be exported, what their arguments could be, and how they fit together. See the prototype pages for implementation details.

probes	args[0]	args[1]	args[2]	args[3]	args[4]	args[5]
gld:::send gld:::receive	pktinfo_t *	NULL	ipinfo_t	illinfo_t *	etherinfo_t *	.
ip:::send ip:::receive	pktinfo_t *	csinfo_t	ipinfo_t *	illinfo_t *	ipv4info_t *	ipv6info_t *
tcp:::send tcp:::receive	pktinfo_t *	csinfo_t *	ipinfo_t *	tcpsinfo_t *	tcpinfo_t *	.
tcp:::accept-* tcp:::connect-*	pktinfo_t *	csinfo_t *	ipinfo_t *	tcpsinfo_t *	tcpinfo_t *	.
tcp:::state-change	NULL	csinfo_t *	tcpsinfo_t *	tcpnsinfo_t *	.	.
tcpf:::send tcpf:::receive	pktinfo_t *	csinfo_t *	ipinfo_t *	tcpsinfo_t *	tcpfinfo_t *	.
udp:::send udp:::receive	pktinfo_t *	csinfo_t *	ipinfo_t *	udpinfo_t *	.	.
udp:::stream-*	pktinfo_t *	.	.	.	.	.
icmp:::send icmp:::receive	pktinfo_t *	NULL	ipinfo_t *	illinfo_t *	icmpinfo_t *	.

Other probes will be added as required.

Arguments

The role of the arguments listed above is as follows:

• pktinfo_t * - packet info; After task 7 in the plan, this will provide packet IDs to allow a packet to be followed as it passes through the network stack. For now, it will use mblk pointers as packet IDs.
• csinfo_t * - connection state info; After task 13 in the plan, this will provide connection IDs to assist associating packets with user-level process IDs. For now, it will use conn_t pointers as packet IDs.
• ipinfo_t * - IP info; basic IP info that is available throughout the stack.
• illinfo_t * - IP lower level info; details about the network interface.
• etherinfo_t * - Ethernet header info.
• ipv4info_t * - IPv4 header info.
• ipv6info_t * - IPv6 header info.
• tcpinfo_t * - TCP header info.
• tcpfuseinfo_t * - TCP fusion details.
• udpinfo_t * - UDP header info.
• icmpinfo_t * - ICMP header info.

Other arguments will be added as required.

Plan

Network stack instrumentation with DTrace will be achieved through two projects, one to create customer-orientated providers, the other for engineer-orintated providers:

• public/stable/customer providers for generic observability
ip:::, tcp:::, udp:::, ...
• private/unstable/engineer interface for code path observability
sdt:::ip*

Customer Provider Plan

The following are rough steps for creating a suite of customer-orientated network providers. Each step may be integrated separately.

1. IP Provider
        ip:::send
        ip:::receive

2. TCP Provider
        tcp:::send
        tcp:::receive
        tcp:::accept-*
        tcp:::connect-*
        tcp:::state-*
        tcp:::retransmit
        tcp:::timeout-*
        tcpf:::send
        tcpf:::receive

3. UDP Provider
        udp:::send
        udp:::receive
        udp:::stream-*

4. IP Provider
        ip:::drop-in
        ip:::drop-out
        (usr/src/uts/common/inet/ip/ipdrop.c)

5. IP Provider
        ip:::request
        ip:::deliver

6. IP packet ids
        ip:::packet-free
        perhaps a new mblk_t member: (uint64_t)mblk_t->b_id;
        future, any suitible uint64_t: (uint64_t)nifty_t->n_id;

7. Socket Provider
        socket:::send
        socket:::receive

8. ICMP Provider
        icmp:::send
        icmp:::receive

9. ARP Provider
        arp:::send
        arp:::receive

10. SCTP Provider
        sctp:::send
        sctp:::receive

11. GLD Provider
        gld:::send
        gld:::receive

12. TCP/UDP connection ids
        /connfs

Dropped from the list

13. DTrace Enhancements
        write() action
        msgdcopy() action

Task 13 would allow a DTrace script to output an RFC 1761 file for reading using snoop or wireshark. A prototype was written, but work has been halted in favour of the other tasks in the plan. Snoop is more suitable for capturing packet data on high speed networks.

Engineer Provider Plan

Many probes have already been integrated into Solaris Nevada for detailed observability of network stack behaviour:

dtrace -ln 'sdt:::ip*' | awk 'BEGIN { OFS=":::" } { print $2,$5 }'
PROVIDER:::NAME
sdt:::ip-ire-del
sdt:::ip-ire-del-origin
sdt:::ip6-physical-out-end
sdt:::ip6-physical-out-end
sdt:::ip6-physical-out-start
sdt:::ip6-loopback-in-end
sdt:::ip6-loopback-in-end
sdt:::ip6-loopback-in-start
sdt:::ip6-loopback-in-start
sdt:::ip6-forwarding-end
sdt:::ip6-forwarding-start
sdt:::ip6-physical-in-end
sdt:::ip6-physical-in-start
sdt:::ip-xmit-incomplete
sdt:::ip-xmit-v4
sdt:::ip-wput-ipsec-bail
sdt:::ip6-loopback-out-end
sdt:::ip6-loopback-out-end
sdt:::ip6-loopback-out-end
sdt:::ip6-loopback-out-start
sdt:::ip6-loopback-out-start
sdt:::ip6-loopback-out-start
sdt:::ip4-loopback-in-end
sdt:::ip4-loopback-in-end
sdt:::ip4-loopback-in-start
sdt:::ip4-loopback-in-start
sdt:::ip4-loopback-out_end
sdt:::ip-xmit-1
sdt:::ip-xmit-2
sdt:::ip4-loopback-out-end
sdt:::ip4-loopback-out-end
sdt:::ip4-loopback-out-end
sdt:::ip4-loopback-out-end
sdt:::ip4-loopback-out-start
sdt:::ip4-loopback-out-start
sdt:::ip4-loopback-out-start
sdt:::ip4-loopback-out-start
sdt:::ip4-dhcpinit-pkt
sdt:::ip4-physical-in-end
sdt:::ip4-physical-in-start
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-forwarding-end
sdt:::ip4-forwarding-end
sdt:::ip4-forwarding-start
sdt:::ip4-forwarding-start
sdt:::ip-newroute-drop
sdt:::ip6-physical-out_start

In particular, the probes added as part of ipf (eg, sdt:::ip4-physical-out-start) form a good model to build upon.

Testing

Ongoing testing of the network providers will be needed for maintaining their accuracy. It is expected to be achieved through two forms of testing:

1. Customer testing. The SUNWdtrt (DTrace Test Suite) package will include basic tests that can be initiated from a single host, as with the other tests in SUNWdtrt. These tests will generate simple network traffic which will be traced, confirming that the probes and arguments work as expected. These tests are expected to be frequently executed by engineers as part of existing ongoing testing of SUNWdtrt, and by customers when troubleshooting DTrace. There are limitations with single host testing, and some code paths such as those that require special hardware (crypto accelerator cards, NICs that support LSO, etc.) will usually not be tested.

2. Engineer testing. Using multiple hosts with a variety of different hardware, tests can be executed by hand by generating traffic, tracing details, and confirming that all values are as expected. This will happen during development of each task from the plan and if/whenever bugs are lodged. A test team at Sun with access to a test farm could take this onboard as an ongoing task.

Solutions

Some of the network obserability problems that this provide will solve include:

• Who is connecting to my web server? (TCP accepts by IP address)
• Which services are busy? (bytes by destination port)
• Which hosts are responsible for network load? (bytes by IP address)
• Are hackers/crackers port scanning my server? (TCP flag matching by IP address)
• Why are outbound connections slow? (Connect latency, 1st byte latency, throughput latency)

Comments

This design has been publically open for comments for several months, and discussed a number of times on the dtrace-discuss mailing list. Time is now running out before these probes will begin to be integrated.