Ты с какой библиотекой линкуешься?
См доки: Q257460
When setting or getting socket options at the IPPROTO_IP level in a C/C++ Winsock application targeted for Windows NT, Windows 2000, Windows 95, or Windows 98, including the correct header and library file in the program project is critical. If the header and library files are not properly matched, setsockopt or getsockopt may fail with runtime error 10042 (WSAENOPROTOOPT). In certain cases, even if the API returns successfully, the option value that you set or get may not be what you would have expected.
To avoid this problem, observe the following guidelines:
A program that includes Winsock.h should only link with Wsock32.lib.
A program that includes Ws2tcpip.h may link with either Ws2_32.lib or Wsock32.lib. Note that Ws2tcpip.h must be explicitly included after Winsock2.h in order to use socket options at this level.
Я попробовал с ws32_2.lib, не работает. C wsock32.lib работает
Хотя совсем не факт, что числовое значение константы, заданное в ws2tcpip.h совпадает с нужным значением, которое используется в библиотеке