| stallion_work | mbreitba: Infiniband is "stupid fast", IPoIB is like wrapping a PDU in bacon and shoving over IB |
|---|
| TommyTheKid | NFS should use RDMA |
|---|
| lewellyn | i'm afraid to open the attachments :) |
|---|
| TommyTheKid | I am not sure about iSCSI |
|---|
| lewellyn | mmmbacon |
|---|
| balrog | lewellyn: what are the attachments? |
|---|
| mbreitba | they can both use RDMA, and will bypass the IP stack if I can get that configured properly, but I want to at least try to get the IPoIB working |
|---|
| TommyTheKid | i want bacon |
|---|
| mbreitba | plus, the RDMA is even less well documented than the IPoIB |
|---|
| stallion_work | mbreitba: be a man and use RDMA |
|---|
| lewellyn | balrog: a .doc and a .pdf |
|---|
| lewellyn | i've apparently been invited to beijing in july |
|---|
| balrog | hmm. |
|---|
| balrog | I see |
|---|
| eviljames | bonsaikitten: I have found my next purchase. Hats off to you! :D |
|---|
| balrog | why are you afraid then? :) |
|---|
| mbreitba | not that IB is well documented at all |
|---|
| wrapster | is there a way i can figure out what internationalization support is provided by default in osol and what can be downloaded later? |
|---|
| rkeene | You won't see the advantages of infiniband over 10GigE if you're using only IP over it. |
|---|
| bonsaikitten | eviljames: really nice hardware, I like them :) |
|---|
| stallion_work | mbreitba: its documented well enough. |
|---|
| lewellyn | because i don't have any ties to the sector, and it was sent with a gmail.com return address over 163's mail servers |
|---|
| mbreitba | the other reason to use IB is that our bladecenter supports it, and the total cost was less than what 10GE would have costed to impliment |
|---|
| balrog | ahh. |
|---|
| balrog | open it in google docs :) |
|---|
| TommyTheKid | I would focus on RDMA, although we had talked about using IBoIP because its fanstastically cheaper than 10GIG (I think) |
|---|
| mbreitba | well, I want to get the simpler protocols working first, then move on to the RDMA |
|---|
| lewellyn | i'd still have to download the attachment ;) |
|---|
| lewellyn | huh. bing says it's likely legit |
|---|
| mbreitba | that, and I don't know that the windows boxes that we're currently using work that well with NFSoRDMA |
|---|
| balrog | I don't see how downloading the attachment could cause issues, unless you open it |
|---|
| mbreitba | here's the situation - we've got a large infrastructure running Windows/Hyper-V |
|---|
| mbreitba | Everything is on iSCSI right now, formatted NTFS |
|---|
| mbreitba | we want to move to either Xen or VMWare (haven't decided which yet) |
|---|
| TommyTheKid | mbreitba: I would check the simple stuff first, dladm show-phys, make sure you have link, ifconfig -a .. make sure its UP, make sure it has a proper IP/netmask/broadcast, make sure there is no firewall (ipfilter?) |
|---|
| mbreitba | want to utilize storage on NFS volumes for data portability |
|---|
| TommyTheKid | unfortunately not sure how any of that maps into nexenta :) |
|---|
| mbreitba | Everything that I've checked looks fine |
|---|
| mbreitba | all of the IB queries show that the interfaces are all connected and up |
|---|
| lewellyn | phone |
|---|
| mbreitba | and the nexenta box is able to ping itself on the IB interface |
|---|
| TommyTheKid | make sure the d0ze side has no firewalls as well |
|---|
| mbreitba | no firewalls on the Win side |
|---|
| TommyTheKid | ofcourse if they can ping eachother... |
|---|
| mbreitba | I'm about 20k deep into this project, and it's really starting to piss me off |
|---|
| stallion_work | mbreitba: well... I wouldnt have high hopes for using just NFS |
|---|
| TommyTheKid | so is your interface ibd0? |
|---|
| mbreitba | ibd0 and ibd1 on the nexenta box |
|---|
| stallion_work | you are better off with a NAS head which supports CIFS/NFS access to the same filesystems |
|---|
| mbreitba | well, the nexenta box would be the NAS head end - I can provide CIFS storage out of it also |
|---|
|
|
| mbreitba | also - more info about the build here : http://www.zfsbuild.com/ - details about the hardware and intentions |
|---|
| TommyTheKid | ok yuck, our IB router is running S10 :( |
|---|
| TommyTheKid | IB router -> system with 10G and IB connecitivy, routing between them |
|---|
| mbreitba | I've already tried using bare OpenSolaris - I actually got the IBoIP working ok on it using 2009.06, but it would frequently just drop the connection |
|---|
| mbreitba | iSCSI over the IPoIB link would reset every 35 seconds (literally, like clockwork) |
|---|
| mbreitba | NFS wouldn't mount, but I'm wondering if that was because it was only offering it over RDMA |
|---|
| mbreitba | IPoIB wouldn't work at all in 134a |
|---|
| TommyTheKid | yea, so I wish I had some great knowledge to impart, I know we are messing with IB in our lab, for the same reason you are, but I don't know where we are with it.. I was just asked for a subnet to route to :) |
|---|
| TommyTheKid | a subnet and a route to it on serverX that is |
|---|
| TommyTheKid | and the reason I grumbled about S10 was nmap isnt there ;) |
|---|
| mbreitba | well, maybe it's time to jump back into OpenSolaris and hack on it from that end.....the Nexenta stuff is a little more black-box'ish and that seems to be complicating things |
|---|
| lewellyn | TommyTheKid: er... yes it is |
|---|
| lewellyn | install the companion cd :P |
|---|
| TommyTheKid | bah |
|---|
| rkeene | Atleast you aren't trying to do IP-over-Fibre-Channel in Solaris 10. It's completely broken and Sun/Oracle refuses to fix it. |
|---|
| TommyTheKid | I am not going to go installing software on their router |
|---|
| TommyTheKid | hehe |
|---|
| TommyTheKid | IBoFC, yikes |
|---|
| lewellyn | your statement is still false ;) |
|---|
| rkeene | IBoFC ? |
|---|
| TommyTheKid | typo |
|---|
| rkeene | Oh |
|---|
| rkeene | fcip(7D) is completely broken |
|---|
| mbreitba | Nexenta actually told me that the IB card that we had wasn't on the OpenSolaris HCL......which was totally wrong, in any event though, I'm hoping to get something running |
|---|
| TommyTheKid | lewellyn: its not there by default, companion CD (/opt/sfw I assume?) is generally hopelessly out of date, so we stopped installing it |
|---|
| mbreitba | how is the CIFS support on OpenSolaris - is it going to work for my Windows hosts until I can get migrated to Xen/VMWare and use NFS? |
|---|
| TommyTheKid | smrt: explain cifs |
|---|
| smrt | File sharing protocol commonly used on Windows. OpenSolaris includes two cifs servers, one builtin to the kernel, and the samba userspace daemon. (See also: cifs server, samba) |
|---|
| lewellyn | TommyTheKid: it may be out of date, but it is part of a "complete" install. hell, most of the os is hopelessly out of date ;) |
|---|
| TommyTheKid | thats true lewellyn :) |
|---|
| TommyTheKid | I used to install it once, centrally and mount it NFS.. but I don't know how long its been since I had that :) |
|---|
| mbreitba | and I'm guessing that if I use CIFS, I can export that via NFS later and Xen/VMWare will still be able to read it? |
|---|
| TommyTheKid | sharesmb=yes? |
|---|
| TommyTheKid | smrt: explain cifs server |
|---|
| smrt | OpenSolaris has a new CIFS (Windows file sharing in SMB) server in the kernel. Good resources for it include http://dlc.sun.com/osol/docs/content/SSMBAG/troubleshooting.html and http://wiki.genunix.org/wiki/index.php/CIFS_Service_Troubleshooting . The CIFS provides better performance and easier administration for CIFS than Samba but Samba provides other SMB services (See also: samba) |
|---|
| lewellyn | mbreitba: if you use the kernel cifs, you can manage nfs and cifs with the same interface |
|---|
| mbreitba | alrighty, I think my answer is to turn back to OpenSolaris then and hack some more on it |
|---|
| mbreitba | also, just out of curiosity, as I didn't find this in my digging - if the NFS server detects RDMA capable hardware, does it only offer NFS over RDMA, or does it offer it up over IP at the same time? |
|---|
| TommyTheKid | I *think* if both ends are on IB, it skips over the IP stack (uses RDMA) |
|---|
| TommyTheKid | but maybe thats because it doesn't offer NFS over IP on IB interfaces,,, I couldn't tell you there ;) |
|---|
| mbreitba | alrighty - thanks for the info - back to hacking on OpenSolaris - maybe I just need a bigger axe ;) |
|---|
| TommyTheKid | mbreitba: good luck :) |
|---|
| TommyTheKid | i dont honestly know why they used S10, probably to keep from scaring the OPS folks :p |
|---|
| TommyTheKid | ... apparently either the routing is messed up or they are firewalling, either way, I am not mucking with their config :) |
|---|
| lewellyn | hrm. has anyone packaged tortoisehg for opensolaris yet? it seems overly logical to have ;) |
|---|
| TommyTheKid | lewellyn: quick question... if I build a package with SFE, can I like scp it to another system and install it? |
|---|
| TommyTheKid | not that I hafe SFE working yet ;) |
|---|
| alanc | lewellyn: GUI's are illogical though |
|---|
| Action: LadySerena is a happy kitty |
| Tonnerre | LadySerena, good! |
|---|
| TommyTheKid | i suppose thats better than beeing a sad salamander ? |
|---|
| LadySerena | At school, I got my pin and senior scrubs today. :D |
|---|
| lewellyn | TommyTheKid: if you build an svr4 package. |
|---|
| lewellyn | alanc: they're still faster than opengrok at getting history ;) |
|---|
| TommyTheKid | lewellyn: otherwise, I can just point at the pkg server that will somehow supposedly be running there on that machine and install it over the network? |
|---|
| lewellyn | and if you're poking around in nautilus, it's fater than opening a terminal and navigating |
|---|
| lewellyn | TommyTheKid: in theory. i've not built ips yet ;) |
|---|
| TommyTheKid | this could be interseting :) |
|---|
| TommyTheKid | interesting even |
|---|
| lewellyn | remember last night how i was asking about pkgtoolrc? ;) |
|---|
| TommyTheKid | hmm, I was bashing my head on the bootstrap script, so I didnt see it :) |
|---|
| lewellyn | i still don't know why you were editing it. |
|---|
| lewellyn | just run the damn thing :P |
|---|
| TommyTheKid | it doesnt work |
|---|
| lewellyn | smrt: explain doesn't work |
|---|
| smrt | Then make it file for unemployment benefits. |
|---|
| lewellyn | heh |
|---|
| TommyTheKid | hah |
|---|
| TommyTheKid | I have an NFS home directory and it doesn't cooperate |
|---|
| TommyTheKid | so I tried to edit the USERHOMEDIR= setting to point at /ws/tm.... and that sorta works, except that it still somehow makes some stuff appear in ~/ |
|---|
| lewellyn | yeah. i keep meaning to patch it to deal with nfs homedirs. |
|---|
| lewellyn | but tomww is quiet and i don't know how out-of-date the current download is vs his local copy |
|---|
| TommyTheKid | :) |
|---|
| TommyTheKid | if he doesn't commit his copy up, then its not there :P |
|---|
| lewellyn | patches are less useful when they're against virtually every line and you're trying to commit them against something fairly different ;) |
|---|
| TommyTheKid | agreed |
|---|
| --- Thu Apr 29 2010 |
Popular searches: 
