Custom networking stack using Linux TAP devices – trouble sending & receiving L2 frames

I’m trying to create a custom network stack for personal reasons and thought I could do an L2 that wasn’t too complicated by using Linux TAP devices and raw sockets. I want to create a “node” which can send to other nodes which are connected point-to-point in L2.

So far, I’m able to create two TAP devices (with C code) and view a packet leaving one of them via the raw socket I created in Wireshark. What I can’t figure out is where the packet actually goes next. I set the MAC address of the ethernet header to be the address of the receiving TAP device (they only have MAC addresses as I want to do IP myself) but I don’t see the packet come up anywhere else.

The main idea here is that I won’t be able to just get the file descriptors (assuming that the nodes are running in separate processes) so I’m trying to communicate between different TAP devices with raw sockets.

Do I need a bridge between the TAP devices, and if so how would I properly integrate that? As in, how would I ensure the packet goes through the bridge to the TAP device (I have a feeling I’m missing something major here). I did try creating one and connecting the two but the packet still didn’t go anywhere.

Here is the code I use to create the TAP device and raw socket:

/**
* Creates and Ups tap intf, creates a raw socket, binds to it
*/
tap_conn_t* start_tap_sock(char* dev_name) 
{
    int tap_fd = tun_alloc(dev_name);
    int raw_fd = socket(AF_PACKET, SOCK_RAW, IPPROTO_RAW);
    if (raw_fd < 0) {
        printf("ERROR CREATING TAP SOCKET, ERRNO: %d\n", errno);
        close(tap_fd);
        return NULL;
    }

    // set the device up by adding IFF_UP flag and make requests 
    // to get if_index and hwaddr
    add_ifflags(raw_fd, dev_name, IFF_UP);
    struct ifreq hwaddr = make_ifreq(raw_fd, dev_name, SIOCGIFHWADDR);
    struct ifreq if_index = make_ifreq(raw_fd, dev_name, SIOCGIFINDEX);

    // create sockaddr_ll and bind tap_conn to it
    struct sockaddr_ll sl;
    sl.sll_family = AF_PACKET;
    sl.sll_protocol = htons(ETH_P_ALL);
    sl.sll_ifindex = if_index.ifr_ifindex;
    sl.sll_halen = ETH_ALEN; // len of hwaddr (MAC addr)
    memcpy(&sl.sll_addr, &hwaddr.ifr_hwaddr, ETH_ALEN);
    int slen = sizeof(struct sockaddr_ll);
    if (bind(raw_fd, (const struct sockaddr*) &sl, (socklen_t) slen) < 0) {
        printf("ERROR TRYING TO BIND SOCKET, ERRNO: %d\n", errno);
        close(tap_fd);
        close(raw_fd);
        return NULL;
    }

    // malloc struct and return
        ...
    return tc;
}

Here is the code I use to send packets:

int send_to_tap(char* remote_name, char* data, int len, tap_conn_t* tc) 
{
    // make ethernet header first
    struct ifreq remote_hwaddr = make_ifreq(tc->raw_sock_fd, remote_name, SIOCGIFHWADDR);
    struct ethhdr ehdr = make_eth(tc->hwaddr, remote_hwaddr.ifr_hwaddr);

    // put header and data into buff
    char send_buf[sizeof(struct ethhdr) + len];
    memcpy(send_buf, &ehdr, sizeof(struct ethhdr));
    memcpy(send_buf + sizeof(struct ethhdr), data, len);
    
    // make sockaddr_ll and send
    struct sockaddr_ll sl;
    sl.sll_family = AF_PACKET;
    sl.sll_ifindex = tc->if_index;
    sl.sll_halen = ETH_ALEN;
    memcpy(&sl.sll_addr, &tc->hwaddr, ETH_ALEN);
    
    int sl_len = sizeof(struct sockaddr_ll);
    int send_len = sendto(tc->raw_sock_fd, (void *) send_buf, sizeof(struct ethhdr) + len, 0, (const struct sockaddr *) &sl, (socklen_t) sl_len);
    if (send_len < 0) {
        printf("ERROR SENDING, ERRNO: %d\n", errno);
        return -1;
    }

    return send_len;
}

I think I’ve seen people say that veth interfaces would work well for this – but not sure how to actually allocate a veth device in C unless I just write this part with bash and ip link or something.

In the end, I could use something like pcap but I figured I’d ask to see if I could get this to work first.