Wireshark makes it a whole lot easier than you make it out to be. It
decodes the traffic. Especially with the fact that it works with an IP
address, yet it didn't work with a dns entry leads me to believe that
wireshark could provide valuable info. I would say at least look at it.
Yes. Wireshark will decode the protocol, but it will not decode the message content, and in this case, there is no problem with network protocol. How do I know? "mount //nas0.tclc.org/d0 /net/nas0/d0 ..." works on all Fedora machines, even when the Windows machines do not.
An investigation of network protocol and message encoding would have been a waste of time even before I solved the problem, but would most definitely be a waste of time now. I solved it in a few hours, and I couldn't have done anything with Wireshark in a few hours. I'd have been digging around in DNS internals, and that would have been pointless because even if I was able to pinpoint some defect, I would not have been able to change anything about that interaction. Wireshark is only going to tell me about problems I can't solve.
The problem is how Windows is handling something about that domain name. Joining the domain enters a domain name for the participating machine in DNS and there is something out of spec in that operation in this case. It is not clear to me if the ASUStor NAS has any way to perform the "Join Domain" operation that could cause this. So, Wireshark might be able tell me something about this, but, frankly, I don't care; that's a discussion between ASUS and Microsoft. If this is where the problem is than the problem is between ASUS and Microsoft, and it will be a monumental waste of time for me to try to debug it. I've already solved the problem!
Knowing what is happening during the "Join Domain" operation, is also pretty pointless, since whatever is going wrong is manifest in the DNS entry, and that has nothing to do with wire protocol or traffic, and I can't do anything about it anyway. A clear understanding of DNS internals would likely provide much more information than looking at network traffic and protocols. And I don't want to do that, either.
Besides, the comand "dir \\nas0.tclc.org\d2" fails on the domain controller with some nonsense about the domain name not existing, so there is no network traffic involved.
But, the most important point is that I solved this:
And I avoided it by naming the ASUStor, "NASTY.TCLC.org" freeing "nas0.tclc.org" for all purposes as before. So, nothing changed beyond spending one throw-away domain name.
I have already reported this to ASUS and they have confirmed the issue and I have a nearly zero cost strategy to avoid the problem. Why do I need to do anything more?